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Abstract. Due to rising product and process complexity, assistance systems for 
humans are increasingly used in both the planning and execution of assembly 
activities or as support systems for reworking. However, developments availa- 
ble so far usually act as isolated solutions. Therefore a continuous dataflow be- 
tween top and shop floor is not given. This article presents a comprehensive as- 
sistance system consisting of a virtual planning environment and various smart 
devices at an assembly workstation. In the virtual environment, the feasibility of 
initially created product condition graphs will be validated and a training of the 
repair scope to be carried out will be done. As a result, a process plan is derived 
from the virtual environment and transferred to an assisted workstation for re- 
working. The repair on the real product is carried out using the created process 
plan so that a continuous data flow is available. Thus the system supports the 
persons involved in the product analysis and work plan creation up to the exe- 
cution of repair activities during rework. 


Keywords: Virtual Assembly, Assembly/Disassembly Planning, Training Plat- 
form, Assistance Systems, Consistent Data Exchange 


1 Introduction 


Planning processes for assembly and disassembly are becoming more and more ex- 
tensive due to increasing product complexity, variant diversity and customer-specific 
individualization options [1, 2]. In addition, manufacturing companies are increasing- 
ly forced to reduce development costs and time. Since assembly volumes represent an 
immense cost factor with regard to the product development process [2, 3] productivi- 
ty, flexibility and efficiency must increase within the scope of the corresponding 
planning. However, this is becoming more and more difficult to achieve as the feasi- 
bility testing of assembly volumes also becomes more extensive. As a result, virtual 
development methods and tools are becoming increasingly important. The three- 
dimensional representation within Virtual Reality (VR) enables the user to visualize 
and interact with the digital product in a realistic way. Time-consuming and cost- 
intensive planning with the aid of physical prototypes thus becomes superfluous. The 
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possible fields of application of VR technology for manufacturing companies are very 
diverse. A classification according to task fields can take place according to: design 
(prototyping), operations management (planning, simulation, training), manufacturing 
processes (production flows, assembly, inspections) [4] and visualization [5]. 

This article focuses on the areas of visualization, training and validation of assembly 
scopes in a virtual environment as well as the consistent data flow to and from the 
VR-Application. Based on that, the use of the results for other applications can be 
enabled. Initially created product status graphs, in which information regarding as- 
sembly and disassembly scopes are stored, are to be validated in a virtual environment 
and adapted if necessary. For this purpose, the assembly scopes are created in a stand- 
ardized platform and transferred to the virtual environment. To achieve a sustainable 
utilization of validated product condition graphs, information can be transmitted to 
other applications e.g. to an assisting workstation. After the initial planning phase the 
virtual environment can be used as a training platform for the further education of 
employees. 

In the following, the current state of research of virtual environments in the field of 
assembly is described (chapter 2). Since the paper focuses the overall data flow be- 
tween VR and other applications, a cognitive assistance system is described as an 
example system which can profit from the validated product and process scopes. An 
overview of the data transfer and the necessary systems is given in chapter 3. The 
technological implementation of the developed VR application, functionalities and the 
realization of the data exchange is described in chapter 4. A summary and outlook is 
given in chapter 5. 


2 State of the art and initial situation 


This chapter highlights the current developments and areas of application in the field 
of virtual environments with regard to assembly planning and production. One focus 
of the paper is on consistent data exchange. Downstream systems should benefit from 
the results of the virtual environment. For this paper a cognitive assistance system in 
the field of rework on the shop floor was used. Therefore the last section of this chap- 
ter discusses the current state of the art of this class of assistance systems. 


2.1 Virtual Environment and Assembly Planning 


Further developments with regard to the performance of software and hardware, as 
well as the possibility of realistic depiction of physical products allow Virtual Envi- 
ronments (VE) to become present in the industrial environment. In the course of in- 
dustry 4.0, VE is a fundamental technology due to the development of new potentials 
and its broad application spectrum. According to Nikolaos et al. [3], a technological 
classification of the VE can be divided into 


e Virtual Manufacturing (production process), 
e Virtual Prototyping (product investigations) as well as 
e Virtual Assembly (assembly scope). 


The traditional procedure for planning assembly and disassembly volumes is strongly 
linked to the expert knowledge of the planner. A fully automatic generation, also with 
the help of powerful CAD programs, fails due to the mathematical formulation and 
utilization of this knowledge. Instead of using abstract algorithms for planning the 
assembly scope, Virtual Assembly (VA) supports the planner within the virtual envi- 
ronment. With regard to the validation of possible assembly and disassembly se- 
quences, various VR applications can be found. One of the first systems developed is 
the Virtual Assembly Design Environment (VADE), in which user interaction, colli- 
sion analyses and assembly tools are provided. Based on this, further VR environ- 
ments exist with regard to validation, visualization and simulation of assembly scopes 
[6-9]. However, virtual environments are often regarded only as isolated solutions. 
The data transfer usually only takes place one way to the VR application, but not back 
again. Thus, the feedback and transfer of changes made to the assembly scope to data 
management systems or further applications is only insufficiently considered. 


2.2 Virtual Training System 


Products are becoming increasingly complex and thus more difficult and expensive to 
manufacture. The training of employees on real objects is therefore only possible to a 
limited extent. New technological developments for supporting training and further 
education are therefore becoming more important [2]. In addition to the applications 
for planning and development, there already exist various VR platforms for mapping 
training scenarios [10-13]. The digital three-dimensional model, assembly as well as 
disassembly sequences are simulated realistically. In addition to the purely visual 
representation within the VR, the increase of the user's performance and learning rate 
could be shown by implementing auditory and tactile feedback [7]. Assembly work 
requiring a high degree of psychomotor skills can be trained thoroughly (learning by 
doing) and cost-effectively within the VR application. The goal of virtual environ- 
ments as learning platforms is always the transfer and recognition of simulated sce- 
narios from user’s in reality [2]. 


2.3 Cognitive assistance systems 


In the manual assembly of complex products, the use of cognitive assistance systems 
(also: digital assistance systems [14]) is suitable for supporting employees. In most 
cases this is done by fading in assembly process information [15, 16]. Extended sys- 
tems have integrated resources for process testing and documentation, so that pro- 
cesses can be checked for their correct execution and logged for traceability purposes. 
This is intended to detect and eliminate errors on site and reduce the return rate. 

The assistance systems are usually operated from the shop floor. A process list (as- 
sembly sequence) is created by the foreman or a comparable employee to enable se- 
cure assembly. Using an example of product assembly, all assembly steps are record- 
ed step by step, together with the associated test characteristics. The next step can 
only be processed if the previous step with the taught-in inspection characteristics has 
been recognized as correct. 


Assistance systems can be used for supporting the reworking of products with many 
variants [17]. Nevertheless, cognitive assistance systems for the rework are so far the 
subject of research and hardly available on the market. 


3 Concept for the creation and validation of assembly 
sequences 


The main system discussed in this paper is a VR-Application for validating assembly 
scopes. Functionalities to visualize and edit those assembly scopes are implemented 
in the VR application. By using a standard exchange format, relevant data can be 
transferred to the VR application and validated. To achieve a flow of information 
from the VR to subsequent systems, the standardized graph tool (SGT) serves as a 
platform for data exchange. In the SGT the initial lists of the assembly order and pro- 
cesses are created already. The SGT was previously developed at ZeMA [18]. Edited 
and validated assembly scopes can be reimported in the SGT and later transferred to 
other systems. In this case a cognitive assistance system to support employees in 
rework where automatically created repair plans provided to the worker allow a faster 
disassembly and assembly of complex, variant diverse products. A summary of those 
three systems is given in Fig. 1. The following chapter provides a detailed description 
of the three systems. 


Virtual Reality Assistance-System 
— e Central platform 
e Visualization of 3D- e Data hub e Information about 
Model f ; ; e Graphical interface for processes 
e Interaction with objects visualization of product, | ® Event driven resource 
e Possibility to edit and process, resource parametrization 
validate assembly orders | e Data import and export e Process checking 
e Training platform via JSON e Documentation 
A 


—>: Data exchange between systems 


Fig. 1 Central characteristics of the virtual environment, standardized graph tool 
(SGT) and assistance system 


4 Implementation of the environment for the creation and 
validation of the assembly sequences and transfer of the work 
plan into reworking 


The following section gives an overview over the previously described systems. Fur- 
thermore, the continuous data exchange between the three subsystems is described. 


4.1 Scope of functions within the Virtual Environment 


The virtual environment offers the possibility of visualization as well as the validation 
and editing of the assembly scope. The displayed product can be assembled and disas- 
sembled step by step within VR-Application according to the planning data created in 
the SGT tool. As shown in Fig. 2(a) the main functionalities are "Move all" (scaling 
and translation of the object in space), "Teleport" (jumping movement of the user, see 
Fig. 2(b)), as well as "Assembly" (evaluation assembly sequence) and "Reset". Color 
coding of successive components or assemblies makes visualization easier. Relevant 
process steps are displayed in a toolbar (see Fig. 2(c)). The correctness of these can 
thus be continuously validated. If it is necessary to edit them, another process can be 
selected and saved. Furthermore, there is the option to adjust the given assembly se- 
quence. Based on the current product status, a subsequently planned component or 
assembly can either be acknowledged or, if desired, replaced by an object determined 
by the user. For this purpose, all of the following components are displayed transpar- 
ently, a chosen component for replacement is highlighted (see Fig. 2(d)). Because of 
this functionality the user is able to step through the parts, so that the current part will 
be colored and selectable. If the assembly scopes are valid, they can be confirmed and 
saved by the planner. Verified assembly sequences are saved in tabular form. This 
makes it possible to identify corresponding product status paths within the JSON file 
and to mark these with corresponding information regarding there validity. The mark- 
ing is done by coloring of the respective edges/processes (see Fig. 3(b)). 


Fig. 2 Functionalities of the Virtual Environment - (a) Menu, (b) Teleport, (c) Visual- 
ization of assembly scopes, (d) Editing process 


Furthermore, validated assembly scopes can be used as training scenarios. Due to 
higher product complexities and more variants, employees need extensive training. 
An alternative to training on real prototypes can be the virtual environment. 


The setup for the virtual environment includes VR glasses, two sensors to detect the 
position of the user, as well as a Leap Motion sensor for the detection of hand move- 
ments and gestures. On one hand, this enables intuitive handling of the application, 
and on the other hand, it allows to assign specific functionalities in different gestures. 
The engine is implemented in the Unity development environment. 


4.2 Development and setup of a planning tool for the creation of graph-based 
assembly sequences 


The standardized graph tool is used to model and depict the relationships between the 
product, the required processes and the required operating resources (see Fig. 3(a)). 
To model and edit the assembly sequences, the user can select and insert elements 
from a configurable selection of product, process and resource templates. In addition 
to the visual adjustments of the inserted elements, the user can dynamically assign 
attributes to the elements. Such attributes can contain product-specific information 
such as product id, included parts of the assembly, manufacturer, etc. Different edges 
can be used to model the relationships (processes) between the product states. The 
name of a specific process which is used to get from product state n to product state 
n+1 can be entered here. To simplify the representation and structuring of elements, it 
is possible to hide different elements and to nest or cluster them hierarchically via 
father-child relationships. Created graph structures can be both exported and imported 
as JSON (JavaScript Object Notation) objects. With implementing the planning tool 
as a Web application, it can be used on a variety of different platforms. 
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4.3 Data Transfer between Virtual Environment and SGT-Platform 


Using the virtual environment requires two input formats. A three-dimensional model 
of the product to be viewed (.obj file), as well as the product states defined in the 
beginning within the SGT platform. The associated assembly volumes are exported 
from the SGT platform on the basis of the standardized exchange format JSON. 

The different possibilities of (dis-)assemble a product result in a branched representa- 
tion of product states within the SGT platform. In order to visualize all of the reason- 
able defined scopes in the SGT platform, each path is exported as a linear lists. There- 
fore the exported JSON file gets parsed and transferred in multiple .csv files, related 
to the number of possible paths. Those .csv-files are loaded one after another into the 
virtual environment. Validating of editing the assembly scopes means, that relevant 
information about assembly order and process will be assigned to the corresponding 
.csv file. After all of the lists are checked, an algorithm will loop over all of the .csv 
files an extract the changes. Therefore the changes can also applied to the JSON file. 
Valid lists are highlighted by a green color, invalid lists are highlighted red. Reim- 
porting the adapted JSON file in the SGT gives the opportunity to quickly under- 
stand/visualize which product state branches are valid and which are not (see Fig. 
3(b)). An overview of the data exchange between the systems is shown in Fig. 4. 


CAD-System Virtual 
= Environment 


Processlists / 
Assemblystructure 


Changed 
Processlists 


Cognitiv 
Assistance-System 


—> Dataflow to the virtual environment 

— Verified Data from the virtual environment 
Fig. 4 Information flow between virtual environment, SGT and subsequent systems 
(e.g. an assistance system) 


4.4 Transferring the routing to rework 


The edited JSON-file enables the use of validated assembly scopes for further appli- 
cations, e.g. the cognitive assistance system for rework. For this assistance system 
various product states that the employee can reach during the repair of a product must 


take into account. Therefore, a product structure is used instead of a classical assem- 
bly priority graph. The product structure is composed of nodes and edges, with the 
nodes representing the respective product status and the edges representing the as- 
sembly processes leading to a status change. In the first step, the employee enters an 
error code into the assistance system via the Co-worker guidance system (see Fig. 5) 
The required repair plan is then output via a database query. This controls the neces- 
sary resources in digital form and provides the employee with the necessary process 
information via the Co-worker guidance system. 

In the technical implementation, the graph is stored in a graph and document data- 
base. The objects are organized in four categories: inform (informing the employee), 
parametrize (parameterizing equipment), validate (process testing) and document 
(documenting processes), since these four areas are covered by the assistance system. 
Each of these objects has a sub-object "address" and "parameter", which contain the 
resource to be controlled as well as the necessary parameters. The graph database 
allows for queries containing graph operations. For example, the shortest path from a 
product state to the target state can be output with a query. 

The validated work plan from the virtual environment is searched by a JSON parser, 
whereby the nodes are written to the database and the processes are appended to the 
edges. The graph structure is set up with the available data and made available to the 
control program for operative operation. 
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Fig. 5 Architecture of the cognitive coworker assistance system 


5 Conclusion and Outlook 


In this article, a VR-supported procedure for the feasibility assurance of reworking 
processes by consistent data exchange was described. The central element of the 
planning was a graph tool that allows the representation of product and process rela- 
tionships and the input of their relevant data. The process relationships are stored as a 
graph together with the corresponding data in a JSON formatted text file. This serves 


as an exchange file for other systems. It is loaded into the VR environment, where a 
planner can start simulating the (dis-)assembly processes. Step by step, individual 
processes are validated and returned to the graph-based planning system as feasible or 
non-feasible. In addition, the virtual environment can be used as a training platform. 
Employees can train on already validated assembly scopes to achieve a faster under- 
standing of the complexity of the product. 

An updated graph can be transferred to other systems. This is shown on the example 
of a rework system, whereby data is transferred for assistance (information and pro- 
cess testing) as well as for process execution and documentation. 

In the future, data from the rework area should be able to be fed back. This would 
allow optimization potentials, uncovered in rework, to be reported in a standardized 
way and introduced transparently in rework. Furthermore, a process list generation 
has to be optimized so that the minimum number of process lists is generated, in order 
to make the validation process effortless and repeatable. 
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Abstract. To manage complexity in modern, highly automated production sys- 
tems, functional simulation for logic validation, known as virtual commission- 
ing, is an important enabler. However, established systems for virtual commis- 
sioning are highly specialized software, which are solely used by simulation 
experts. In this contribution, we present a system for intuitive, user-integrated 
virtual commissioning. Using virtual reality, we allow for a first person experi- 
ence of the simulation at runtime. By implementing a stereo-camera-based hand 
tracking system, we facilitate intuitive manual interaction with the simulation, 
including physical manipulation. Furthermore, we also integrate common hu- 
man-machine interfaces, like touch panels, which are directly streamed from the 
industrial controller, offering the operator comparable interaction experience as 
with the real system. We conduct a user study quantifying the intuitiveness of 
interaction for defined tasks of our system compared to a standard simulation 
setup. 


Keywords: Virtual Commissioning, Virtual Reality, Motion Capturing. 


1 Introduction 


Increasing complexity in functional control logic of production system drives the need 
for a thorough testing of these algorithms. Virtual commissioning (VC), a method for 
testing programmable logic controller (PLC) functionality by connecting it to a simu- 
lation emulating the real system, is an important tool in this context [1, 2]. However, 
most production systems still require frequent interaction with human operators for 
value creation, setup or adaption, maintenance or fault remedying. This human factor 
is thus critical for a comprehensive simulation of system behavior and function. The 
active integration of humans into VC environments is hence an important research 
topic. In the next section, we highlight related work for human integration into simu- 
lations and VC of production systems. Based on this, we derive the research gap to be 
addressed in this contribution. An overview of the abstract solution structure includ- 
ing main components and functions is given before a real implementation is de- 
scribed. Based on a robotic handling use-case incorporating many common automa- 
tion components, we conduct a user study focusing on the interaction capabilities of 
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our system. We describe and discuss the results of this study before summarizing our 
research and giving an outlook to future research areas in this field. 


2 State of the Art 


2.1 Related work 


Virtual commissioning is defined as the development and testing of functional logic 
on a simulation model for later deployment in a real system [1]. For this, a virtual 
commissioning setup consists of one or more simulation tools and PLCs connected to 
it. Each PLC can be emulated in separate PLC simulation tool or consist of a hard- 
ware PLC connected to the simulation system. For special system components, like 
industrial robots, the integration of further controllers may be necessary to ensure 
realistic behavior.[2] 

The integration of humans into such simulation systems is still a research focus. This 
is mainly due to the inherent non-determinism of human behavior, that makes a com- 
prehensive simulation challenging. However, as human interaction with (partially) 
automated systems is common, it is also a vital part of virtual commissioning. 

Especially in the field of human robot collaboration (HRC), the simulation of hu- 
mans has been extensively researched. Some approaches use ergonomics simulation 
tools to offline program the behavior of a digital human model (DHM) [3, 4]. While 
this method can be used for workstation design and ergonomics improvements, real 
interaction is hardly implementable. 

Other approaches focus on a live integration, using different forms of motion cap- 
turing systems (MCS) to map a real human’s behavior onto a DHM. De Giorgio et al. 
[5], Kim et al. [6] and Matsas et al. [7] each implement robot models into a Unity®- 
based environment which is used for VR visualization. These environments are de- 
signed as standalone applications for each system to be simulated. Also, system and 
robot behavior are implemented through scripts in the simulation. 

Dahl et al. [8] and Metzner et al. [9] implement VR directly into a virtual commis- 
sioning tool for robotic applications, Tecnomatix Process Simulate®, to enable live 
human integration. Motion capturing is implemented using gaming controllers or full- 
body tracking. The simulation tool itself does not support physics or general mecha- 
tronics simulation. 

Metzner et al. furthermore present an approach to directly map the operator’s 
hands, as the most relevant channel of interaction, into a physics-integrated mecha- 
tronics simulation tool [10]. In this approach, however, no VR technology is imple- 
mented, decreasing intuitiveness of interaction. Allmacher et al. use a similar ap- 
proach, implementing the user including his hands into a Unity®-based VR simula- 
tion of an automated guided vehicle [11]. 


2.2 Research Gap and Contribution 


In the following section, we derive evaluation criteria and benchmark the described 
approaches to derive a research gap. 
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Current research in the field of human integration to VC systems mostly focuses on 
realistic first person visualization of the simulated system. These systems are however 
mostly standalone applications developed in gaming engines [5—7, 11]. This not only 
greatly decreases re-usability of components and functional logic; it also increases 
implementation efforts and prevents realistic system behavior. As actual logic and 
motion controllers, like PLCs and robot controllers, are mostly only imitated through 
customized behavior programming in the gaming application, these systems also do 
not fulfil the requirements for virtual commissioning. 

Furthermore, the sole visualization of the system is not sufficient to enable realistic 
interaction. The realistic mapping of the operator’s actions in the simulation is also 
vital. Some contributions integrate physical gaming controllers from the VR system to 
the virtual scene, allowing for an abstract interaction [6, 8]. Other contributions real- 
ize the mapping of the operator’s hands through motion capturing systems, finding 
that this increases intuitiveness of interaction [10, 11]. These contributions however 
either base on gaming setups or lack VR visualization. 

The research gap is thus the live integration of a human operator, mainly his hands, 
into a realistic VR visualization of a simulation controlled by at least one PLC 
through motion capturing. To allow interaction similar as with the real system, grasp- 
ing and other physical interaction with parts is necessary. Furthermore, the pressing of 
buttons and the interaction with HMI panels is also important. As virtual commission- 
ing is used to test real control logic, the integration of PLCs and, optionally, robot 
controllers and other specialized controllers is necessary. This is complemented by 
the requirement that device behavior in the simulation is realistic and that the control 
algorithms can thus be transferred to the real system. To allow scalable use for differ- 
ent systems, a compatibility with standard CAD systems is necessary to avoid redun- 
dant modeling efforts. As displayed in Table 1, no current system fulfills these re- 
quirements. We thus present a system that aims to combine the approaches presented 
in [5-7] with interaction capabilities from [10] while maintaining CAD/PLM compat- 
ibility. 


Table 1. Evaluation of human-integrated VC approaches 


[3, 4] [5-7] 


Realistic VR visualization es 


Physics simulation -- 


User integration (Motion Capturing) -- 


Interaction capabilities 


(Physical Interaction, Buttons/Switches, HMI) ~ | 0 | | 
Realistic PLC/device behavior - $+ 0 | i 0 


Realistic peripheral device/robot behavior +30 | i 4 


Transferability of algorithms to real application - = 
CAD/PLM compatibility + i =- | : = 
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3 System Concept 


The system is based on the real-time mapping of a simulation environment to a VR 
scene while also translating inputs and interaction from the user back into the simula- 
tion or PLC. Thus, the user is tracked through a motion capturing system during the 
simulation. We consider the hands as the predominant interaction channel from hu- 
man to machine. To allow realistic physical interaction, a simulation of physical be- 
havior, especially rigid body physics, such as gravity and collisions is required. As 
human-machine interaction is often not only done through physical manipulation of 
parts, buttons and switches but via HMI panels, a corresponding representation in VR 
is also necessary. To ensure PLM/VC compatibility, the industrial simulation envi- 
ronment is the center component. Physics simulation and interaction capabilities 
available in the VR engine are not used, as these effects would only affect the visuali- 
zation rather than the simulation. Hence, the simulation environment is the single 
source of truth considering system states as well as object and hands placement. 


4 Implementation and Results 


4.1 System Setup 


The implementation integrates a Leap Motion Controller® (LM) MCS mounted on an 
HTC Vive® HMD. The LM uses an active infrared stereo camera for hand recogni- 
tion and tracking. The HMD is tracked via infrared laser-emitting beacons, which 
trigger sensors on the HMD and allow pose estimation. For fast movements, an iner- 
tial measurement unit is also integrated. The simulation is modelled in the Mechatron- 
ics Concept Designer® (MCD), a VC tool in the industrial CAD system Siemens 
NX®, which also features physics simulation capabilities, capable of covering many 
mechatronics and automation-related simulation tasks [12-14]. The VR rendering of 
the scene is performed in the Unity® Engine. It is hence open for combination with 
different simulation tools or frameworks such as ROS. PLCSIM Advanced® is used 
to emulate a SIMATIC S7® PLC, which is connected to an emulated WinCC HMI in 
TIA Portal®. 

Two computers are used for the evaluation: a standard CAD laptop workstation 
(Workstation 1) running the mechatronics simulation and PLC emulation, and a VR- 
ready workstation (Workstation 2) running the VR environment and processing track- 
ing data (Fig. 1). This setup enables the reusability of the concept for multiple simula- 
tion experts without the need of individual VR-ready computers, since any computer 
capable of running the VC can be combined with a VR-ready workstation. 


Fig. 1. Hardware experiment setup including the virtual WinCC HMI (1), MCD simulation 
environment (2), VR visualization (3) and a HTC Vive® HMD with mounted Leap Motion® 
Controller (4) 


The position data of each rigid body in MCD is streamed to the VR scene in Work- 
station 2 via UDP. The tracking data of the HMD is streamed to Workstation 1 via the 
same protocol, where a virtual LM sensor is placed accordingly in the simulation. 
This function is necessary to compensate the kinematic dependency between the 
HMD and the MCS. The hand tracking data, transferred via a TCP Web Socket con- 
nection, is then used to place a virtual hand skeleton model, described in [10], relative 
to the virtual sensor. The hand links are modelled as collision bodies, allowing for 
physical interaction with other simulation components. Physical and gesture-based 
gripping functionalities, described in [10], are implemented. As the skeletal visualiza- 
tion of the hand has proven uncanny to some users in VR, an overlay of a more hu- 
manoid hand model in the VR environment is implemented, see Fig. 2. 


Fig. 2. Simulation (a) and VR environment (b); Virtual hands (1) in skeletal form (left) and 
with VR overlay (right), gripping object (2) and virtual LM sensor (3) 


The HMI is streamed into the VR environment through a virtual network connection 
(VNC) as an animation when the user turns the left palm towards his face (Fig. 3). 
The hand tracking data of the right index finger is then used to project a pointer into 
the WinCC environment, allowing for interaction with the virtual touch display. 
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Fig. 3. Tracking and visualization of the user’s hands; a) skeletal representation in MCD and 
virtual LM sensor, b) combined visualization in VR, c) HMI touch functionality through VNC 


4.2 Evaluation Use-Case 


The use-case for the evaluation is a combined assembly and screw-driving cell. Two 
components are transported via separate conveyors towards an articulated robot arm. 
Mechanical baffles ensure a correct positioning of the components. The robot arm 
assembles the components at a processing position. Via conveyor, the assembly is 
transported to an automated pick-and-place screw driver, mounted to a portal axis, for 
final processing. The finished assembly is then transported out of the station on the 
third conveyor, see Fig. 4. The station is controlled through a PLC, monitoring several 
laser sensors in the station as well as equipment status signals. The entire station, 
including sensor and actuators, is modelled in MCD. An HMI is used to set the status 
of the system and the robot separately, as well as monitor signals and performance. 
The machine setup was not designed for this use case. An already existing simulation 
model is used for the validation of our system to prove re-usability of existing models. 
We choose this setup as it unites common automation components, including convey- 
ors, linear axes, automated tools, an articulated robot arm, fixtures and feeders, and 
thus highlights the universality of the approach. 
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Fig. 4. Evaluation use-case overview: a) WinCC® HMI; b) PLC emulation software PLCSIM 
Advanced®; c) Simulation environment featuring a lower casing supply (1), an upper casing 
supply (2), an articulated robot arm (3) for handling and an automatic screw driving unit (4) 


4.3 User Study 


The intuitiveness and functionality of the system is tested through a user study. Our 
system is benchmarked to a mouse-keyboard (KM) setting. The task of the operator is 
to activate the station and the robot through the HMI panel. Then, new parts to be 
processed have to be placed on the two inbound conveyors in specific orientation. The 
parts are located on pillars next to the station. The goal of the simulation is to deter- 
mine, whether the designed mechanical baffles enable a robust re-orientation of the 
parts to allow robotic handling. To this end, an automated generation of part instances 
in the simulation cannot fully satisfy the requirement to map the operators influence 
on the system. Furthermore, the logic control functionality as well as robot programs 
are also to be verified through VC. As the station consists of multiple sensors, differ- 
ent actuator types including a robot arm, and mechanical baffles, it represents many of 
the aspects commonly modelled in VC. 

Intuitiveness is rated on an ascending scale from 1 to 5. A total of 30 participants 
took part in the study. 12 participants had no previous experience with the simulation 
environment, 27 had no experience with the VR environment and none had used a 
Leap Motion® controller before. The results of the study are displayed in Fig. 5. The 
KM setup is rated between | and 4 with a mean score of 2, while the proposed VR 
approach is rated between 3 and 5 with a mean score of 4.5. 

It is hence concluded that the proposed concept outperforms the common KM set- 
up in terms of intuitiveness. Furthermore, even though 12 participants had no experi- 
ence with the simulation tool itself, high ratings are prevalent. This hints at a possible 
use of the system for training purposes, as it is seemingly well usable also for users 
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with no previous simulation experience. Even for experienced users of the simulation 
tools with no previous VR experience, the system still achieves better ratings. 


Mouse & 
Keyboard 


m VR-Integrated 
Setup 


Frequency 


1 2 3 4 5 
(lowest) Rating (highest) 


Fig. 5. Frequency of intuitiveness ratings in the user study comparing the common KM setup 
with the proposed VR-integrated approach 


5 Conclusion and Outlook 


This contribution describes a method to integrate a human operator into a virtual 
commissioning system at real time. Using VR and motion capturing, an intuitive in- 
teraction with the virtual system is enabled. An implementation of the concept for a 
robot-based production system based on industrial simulation software is shown. The 
setup is compared to a KM setup in a user study, indicating a significantly higher 
usability and hinting at new potentials for application in operator training. 

In future research, the integration of a full body tracking system will be examined. 
Furthermore, a field study to assess the potentials of using this system for operator 
training should be conducted. As the VR engine used in our setup is also compatible 
with established augmented reality devices, further use-cases basing on this technolo- 
gy, like real-time control of a physical system through manual interaction with a digi- 
tal representation, can be explored. 
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Abstract. This work assesses the accuracy of mono and stereo vision-based 
marker tracking on the Microsoft HoloLens as a representative of current gener- 
ation AR devices. This is motivated by the need to employ object tracking in 
industrial AR applications. We integrate mono and stereo vision-based marker 
tracking with the HoloLens. A calibration procedure is developed that allows us- 
ers to assess the accuracy of the calibration alignments by walking around the 
virtual calibration target. This can be generally applied when calibrating addi- 
tional tracking systems with ready-made AR systems. Finally, the accuracy that 
can be achieved with the developed system is evaluated (comparing the influence 
of different parameters). 


Keywords: AR, industrial applications, OST-HMD, HoloLens, marker track- 
ing, stereo vision, calibration, accuracy. 


1 Introduction 


Industrial applications of Augmented Reality (AR) have several domain-specific re- 
quirements such as the accuracy of tracking, robustness, safety, ergonomics and ease 
of use. One especially important requirement is the accurate tracking of real objects in 
a workplace, for example, to place virtual work instructions on a workpiece. Possible 
applications include assembly, training, but also more complex applications like AR- 
assisted programming of industrial robots. 

The inside-out tracking methods of current generation AR devices such as the Mi- 
crosoft HoloLens or the Magic Leap One allow to accurately place virtual objects rela- 
tive to the world. When the user moves around, the objects stay in their assigned posi- 
tion with little jitter and high accuracy [1]. However, the inside-out tracking in and of 
itself doesn't allow application developers to place virtual objects relative to real objects 
in the scene. In order to achieve that, dedicated object tracking is required. It can be 
accomplished by choosing from different AR tracking techniques available: sensor- 
based (magnetic, acoustic, inertial, optical and mechanical), vision-based (feature- 
based or model-based) and hybrid techniques based on sensor fusion [2]. 
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All of these techniques have their strengths and weaknesses. Vision-based tracking 
requires no additional instrumentation, since the aforementioned AR devices already 
have cameras integrated into them. This allows for very low-cost solutions that can 
offer high accuracy. Its disadvantages are the limited computing power of the AR de- 
vice, limited robustness, the reliance on the line of sight and lighting conditions. [2] 
Fiducial marker tracking is a simple variant of vision-based tracking that offers rela- 
tively high robustness at the cost of the effort to place the fiducials accurately. It is 
readily available through commercial and open-source software. Microsoft endorses 
the use of Vuforia! with the HoloLens. Open-source solutions such as ArUco? and AR- 
Toolkit? are alternatives that can be tailored to the application's needs more freely. 

The goal of this paper is to assess the accuracy of mono and stereo vision-based 
marker tracking on the HoloLens as a representative of current generation AR hard- 
ware. Our contributions include: 


e Integration of stereo vision-based marker tracking with the HoloLens 

e A manual calibration procedure that allows users to assess the accuracy of the cali- 
bration alignments by walking around the virtual calibration target, which can be 
applied for ready-made AR systems like the HoloLens 

e Examination of the accuracy that can be achieved with this setup (comparing mono 
vision to stereo vision) 


In Chapter 2 the theoretical background and related research is presented. Following, 
in Chapter 3 the methods that are used to conduct the assessment of the accuracy are 
described. In Chapter 4 the results are presented and discussed. Finally, in Chapter 5 a 
conclusion is drawn and an outlook on future work is presented. 


2 Related Work 


Many prototypes of applications based on the Microsoft HoloLens employ either a) 
manual registration by the user [3] or b) marker tracking [4-6]. Option a) offers the 
advantage of being easy to implement, and can produce accurate registration. Option 
b) offers the advantage of not requiring user interaction and can dynamically track mul- 
tiple objects. 

Marker tracking uses computer vision algorithms to estimate the pose of a marker 
based on camera images. It can be abstracted to the Perspective-n-Point (PnP) problem 
which describes the task of estimating the pose of a set of n points (with n = 4 for 
square markers, each corner of the marker representing one point). In a first step, the 
corner points have to be extracted from the raw camera image. In a second step the pose 
estimation is calculated. [7] There are different algorithms available that solve the PnP 
problem [8]. ARToolkit 5 uses the Iterative Closest Point algorithm for mono camera 
marker tracking and Bundle Adjustment for stereo camera marker tracking. ArUco uses 


! Vuforia: https://developer.vuforia.com/ 
2 ArUco: https://sourceforge.net/projects/aruco/files/ 
3 ARToolkit 5.4 snapshot, project discontinued: https://github.com/artoolkit/ARToolKit5 
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homography estimation followed by an iterative Levenberg-Marquardt optimization. 
There are more advanced algorithms for the special case of 4 coplanar points that can 
avoid local optima [9]. This is implemented by ARToolkitPlus®. There are several stud- 
ies which characterize the accuracy of different marker tracking solutions [7, 10-12]. 

AR systems generally need to be calibrated at least once in order to achieve regis- 
tration of real and virtual content. Calibration is aimed at eliminating systematic errors 
as well as calculating the correct viewing parameters. There are calibration procedures 
for the different subsystems, e.g. camera calibration, and the system as a whole needs 
to be calibrated as well. For Optical See-Through Head Mounted Displays (OST- 
HMDs) there are manual, semi-automatic, and automatic procedures [13]. Manual pro- 
cedures such as the classical SPAAM [14], Stereo-SPAAM [15] and SPAAM2 [16] are 
based on the user creating so called correspondences from manual alignment of real 
and virtual content. Based on multiple of those correspondences, the viewing parame- 
ters are then calculated. 

Most of those classical calibration procedures assume that the AR display is uncali- 
brated, however, with modern devices such as the HoloLens the system as a whole 
comes pre-calibrated (except for a quick IPD calibration). Qian et al. [17, 18] use this 
fact as a motivation to present a new manual calibration approach. A linear calibration 
model that maps from some 3D-tracker space (which can be any tracking system, ex- 
ternal or head-mounted) to the 3D-virtual space (the virtual 3D-space created by the 
display) is proposed. It is calculated from 3D-3D-correspondences created by the user. 
One of two scenarios being evaluated uses marker tracking based on the RGB camera 
of the HoloLens. The results indicate a somewhat limited accuracy in the depth axis of 
the marker tracking camera, when compared to the other two axes. This depth accuracy 
could be improved by employing stereo vision because it yields a higher information 
content. Besides, the calibration relies on the depth perception of the display. Our own 
preliminary experiments have shown that the depth perception of the display can be 
quite misleading. Furthermore, the working space is confined to arm’s length which 
doesn’t fit the requirements for object tracking in most industrial applications. There- 
fore, we modify the presented calibration procedure, as we believe it is generally well 
suited for modern AR devices where systems come pre-calibrated. 


3 Method 


The notion of accuracy can be divided into two aspects, 1) trueness and 2) precision. In 
the scope of this work, the focus will lie on the trueness. Striving for high trueness (i.e. 
small systematic error) is what makes a calibration procedure essential. The precision 
of a tracking system can be improved by employing filtering techniques such as aver- 
aging or Kalman filtering, which isn’t considered in this work. 


4 ARToolkitPlus, project discontinued: https://github.com/paroj/artoolkitplus 
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First, the concepts for a) the integration of the marker tracking into the HoloLens, b) 
the calibration procedure and c) the evaluation procedure, need to be outlined. The con- 
cepts need to take the industrial application’s requirements as well as the properties of 
the used AR system into account. 


3.1 Marker Tracking Integration 
The requirements for object tracking in an industrial application are chosen as follows: 


e The object tracking needs to have a high accuracy in all three cartesian axes 

e It needs to respect the hardware restrictions of a ready-made mobile AR device. 

e The object tracking needs to be robust against losing the line of sight to the marker 
because, in real-world applications, markers can be occluded or the line of sight can 
be lost completely for some time. In both cases we assume that it is desirable to use 
the last known position before the line of sight was lost. It is expected that within 
some time the line of sight will be restored and the marker tracking can start tracking 
the current marker position once again. 

e The working space needs to range from 60 cm to 150 cm (distance from the marker 
to the tracking system, see Fig. 1, left), so that objects at a medium distance can still 
be tracked accurately. This can be tailored to a specific application’s needs later on. 


We propose a method where, first, the marker tracking determines the pose of an object 
relative to the camera of the HoloLens. It is then combined with the information from 
the built-in inside-out tracking, so that the marker pose with respect to the device’s 
world coordinate frame can be computed. A virtual object could then be placed relative 
to the marker in the world coordinate frame, using the inside-out tracking. This is re- 
peated in small time intervals, so an up-to-date pose of the object is always available. 
This method offers the advantage that even when losing the line of sight, the procedure 
can still track the pose of the marker where it was last seen, which addresses one of the 
core requirements. The disadvantage of this approach is the reliance on the inside-out 
tracking. Furthermore, only the last known position can be tracked if the line of sight is 
lost. If the marker moves in the meantime the tracking can be severely inaccurate. We 
propose to deal with this per use-case, e.g. handling the loss of the line of sight, or 
employing a completely different tracking method. 

The marker tracking is implemented using ARToolkit 5.3 and Unity 2017.1 game 
engine as a framework. The tracking target is a 3D cube with markers of 10 cm width 
on each face (see Fig. 1, right). For mono vision the RGB camera (resolution of 
896x504) in the center of the front of the HoloLens, for stereo vision the central two of 
the four environment understanding cameras (resolution of 480x640 each, only availa- 
ble in “research mode”) of the device are used. 
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Fig. 1. Left: Working space, right: Marker cube 


32 Calibration procedure 


In the calibration procedure the user needs to create a set of correspondences by align- 
ing a virtual cube with a marker cube manually, following a predefined protocol. It 
bases on the procedure described by Qian et al. [17]. We give a brief summary of their 
procedure first, and then explain the modifications we developed. 

In Qian’s procedure a calibration model that maps from some 3D-tracker space to 
the 3D-virtual space (the virtual 3D-space relative to the display) is calculated based on 
3D-3D-correspondences collected by the user. A linear transformation T € R*** is as- 
sumed as a calibration model. It maps points q1, ...,dn € R? from the tracker space to 
points pı, ..., Pn € R? from the virtual space: 


Pi =T* Qi (1) 


Three different models with varying degrees of freedom (dof) are used, namely isomet- 
ric (6 dof), affine (12 dof), perspective (15 dof). 

The correspondences are collected as follows: The points p; are several points that 
are fixed with relation to the display space. The points q; are the corresponding meas- 
urements from the tracking system, where the tracking target is a hand-held cube on a 
stick. To collect correspondences the user is required to coordinate hand and head 
movements in order to align p; on the virtual cube with q; on the marker cube, which 
is repeated multiple times. 

We modify this procedure so that the points p; are gathered based on a virtual cube 
that can be positioned freely in the device’s world coordinate frame by the user via 
keyboard input. The marker cube is placed on a table. This makes use of the system’s 
ready-made property and allows users to walk around the virtual calibration target 
freely while adjusting it to exactly align with the marker cube (see Fig. 2, left). Once 
the two cubes are aligned the user can signal the system to collect a correspondence. 
The correspondences are collected at locations that are combinations of different angles 
(a = —45°, 0°, +45°) and distances (d = 60 cm, 90 cm, 120 cm, 150 cm) between 
the HMD and the marker, 12 in total, so as to have a good coverage of the chosen 
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working space (see Fig. 2, right). Additionally, for each correspondence 12 individual 
measurements by the tracking system are carried out and averaged. 

This modification has the advantage of allowing for a larger working space beyond 
arm’s length and being able to assess the proper alignment of the cubes more easily, 
since it doesn’t rely on the depth perception of the display which can be very misleading 
from our experience. Furthermore, this eliminates the need for head-arm coordination 
which improves ergonomics and accuracy. 

In order to calculate the calibration model T we minimize the sum of squares of the 
absolute value of the residual errors over all correspondences (p;, qi): 


E = Yiallp: -T > gill (2) 


Pı u 


Fig. 2. Left: Aligning p, with q4, right: Calibration Setup 


3.3 Evaluation Concept 


For the evaluation we adopt Qian’s [17] calibrate-and-test procedure. The quality of the 
computed model will be assessed by collecting a new set of another 12 correspondences 
and testing how the previously calculated model fits each data point of the test data set. 
This is done by calculating the residual error e: 


e=p-T'q (3) 


This residual error is used as a metric for the trueness post-calibration. The new corre- 
spondences are collected in locations that are combinations of different angles (a = 
—45°, 0°, +45°) and distances (d = 75 cm, 105 cm, 135 cm, 165 cm). Note that the 
distances differ from those from the calibration phase, so as to assess if the calibration 
model works as well in different locations. 

It needs to be noted that if the display space possessed large errors when compared 
to the real world, the calculated residual error would be misleading, since it is expressed 
in relation to the display space and not the real world. It is assumed that the error of the 
display space is small, since the HoloLens is a pre-calibrated AR system. 

The advantage of the chosen evaluation approach bases on the fact that manual align- 
ments can closely represent an ideal alignment as seen by the user. If the user cannot 
distinguish the position of a real object and a superimposed virtual object, he has an 
optimal experience. 

Finally, the results obtained with mono vision are compared with those obtained with 
stereo vision. 
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4 Evaluation 


4.1 Results 


For the evaluation, the calibration procedure is conducted and subsequently evaluated 
as outlined in the evaluation concept. These two steps are repeated three times. That 
way it is assured that the calibration procedure yields reproducible results. The param- 
eters that are evaluated are chosen as follows: 


e Camera setup: Mono or Stereo 

e Axis: x, y, z or absolute (z represents the depth direction with respect to the display 
space; x and y are perpendicular to that direction) 

e Calibration Model: Isometric, Affine or Perspective 


For each configuration (e.g. “z, Stereo, Affine”) the trueness over all the 12 correspond- 
ences (distributed across the working space) and all three runs (i.e. 36 data points per 
configuration in total) are averaged. 

Fig. 3 shows boxplots for the absolute residual errors of each configuration. With 
mono vision the absolute residual is (12.0 + 6.7 mm) with the isometric model, (3.5 + 
2.0 mm) with the affine model and (3.6 + 1.8 mm) with the perspective model. With 
stereo vision the absolute residual error that can be achieved is (40.3 + 34.2 mm) with 
the isometric model, (20.2 + 17.4 mm) with the affine model and (19.7 + 13.9 mm) 
with the perspective model. 
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Fig. 3. Boxplots for the absolute value of the residual errors 


Table 1 and Table 2 show a more detailed overview of the trueness that stereo vision 
and mono vision can achieve with the setup developed in this work. 


Table 1. Residual error (mean and standard deviation) by model and axis (mono vision) 


x (mm) y (mm) z (mm) absolute (mm) 
Model mean std mean std mean std mean Std 
Uncalib. 10.2 2.3 19.2 1.2 37.1 12.7 43.7 10.3 
Isometric -0.1 2.4 -0.4 3.7 -4.1 12.5 12.0 6.7 
Affine -0.8 1.3 -0.2 1.3 0.0 3.6 3.5 2.0 
Perspective | -0.7 1.3 -0.1 1.2 0.0 3.6 3.6 1.8 
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Table 2. Residual error (mean and standard deviation) by model and axis (stereo vision) 


x (mm) y (mm) z (mm) absolute (mm) 

Model mean std mean std mean std mean Std 
Uncalib. -28.0 4.2 -223.1 | 62.3 -4.0 35.8 227.7 62.3 
Isometric -2.8 4.0 2.7 5.8 -34.6 39.3 40.3 34.2 
Affine -2.1 2.4 4.8 3.8 -18.2 18.2 20.2 17.4 
Perspective | -2.2 2.9 4.6 3.1 -16.9 15.9 19.7 13.9 


4.2 Discussion 


First, it has to be discussed how the values for the mean and standard deviation can be 
interpreted. As stated above, 36 data points are combined for each configuration. The 
mean gives a summary of the overall trueness of each configuration across all locations. 
The standard deviation has to be interpreted with some care. A varying trueness be- 
tween the different locations would contribute to the standard deviation, as well as a 
fluctuation in the three runs. However, these two factors can’t be differentiated with the 
data presented. 

Bearing that in mind, the results indicate that mono vision yields clearly superior 
trueness when compared to stereo vision. This is attributed to the fact that the mono 
camera has got a higher resolution than the two stereo cameras individually, which 
plays a big role especially at large distances between the HMD and the marker that were 
included in the experiments. Furthermore, a stereo vision system is geometrically more 
complex so it apparently can’t be calibrated sufficiently with a linear model, let alone 
an isometric one. This has to be investigated further. 

It is concluded that the presented calibration procedure yields good results for sim- 
pler systems like the mono vision system. It is notable that the trueness in z-direction 
is good, which we attribute to the design of the calibration procedure that is not reliant 
on the depth perception. 

It is observed that the affine and perspective models yield better results than the 
isometric model for both mono and stereo vision. All in all, there is no apparent differ- 
ence between affine and perspective, which is why we prefer to use the affine model 
with less degrees of freedom in a future application. 

With mono vision the mean trueness of the absolute value is at 3.5 mm with the 
affine model which satisfies our requirement for a high accuracy. However, we need to 
take a closer look at the trueness in different locations in the future, since the standard 
deviation of the trueness indicates fluctuation among the different locations and runs. 
The residual errors are generally thought to be due to non-linear errors that cannot be 
compensated by a linear model. 


5 Conclusion 


We conclude that the proposed integration concept and calibration procedure are feasi- 
ble for object tracking in industrial applications. The accuracy that could be achieved 
by mono vision is sufficient for many potential applications. 
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However, stereo vision-based marker tracking on the HoloLens yields no improve- 
ment over using mono vision in the working space that was examined. 

We have achieved 1) the integration of mono and stereo vision-based marker track- 
ing with the HoloLens, 2) the development of a new manual calibration procedure that 
allows users to assess the accuracy of the calibration alignments by walking around the 
virtual calibration target which can be applied for ready-made AR systems like the Ho- 
loLens and 3) an assessment of the accuracy of both mono and stereo vision-based 
marker tracking. 

In the future we want to further examine and improve the accuracy of mono and 
stereo-based marker tracking with the HoloLens by fine-tuning different parameters 
and analyzing the data we collected in more depth. Furthermore, we want to explore 
how marker tracking can be integrated into prototype applications. 
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Abstract. A representation of the world as a 3D model is a common 
necessity in robotics and automation. In previous work, we developed a 
concept to generate boundary representation (B-Rep) models from mul- 
tiple point clouds using a hand-held depth-camera and to register them 
without a prior known pose. During the online reconstruction, properties 
of the sensor and the system (like noise) lead to small holes in the B- 
Rep. To prevent tedious post-processing, holes should be closed during 
the reconstruction. Our goal is to automatically close identified holes. 
However, not every hole can be closed automatically, as it may be unrea- 
sonable. For this case we develop a visual indication for the user, so he 
can close the hole by recording another depth image. In an experimental 
validation, we conclude the usefulness of the addition to the system. 


Keywords: Reconstruction, Volumetric models, Visualization, CAD 


1 Introduction 


To generate a 3D representation of the surrounding world is a necessary skill 
needed in robotics, automation, and their applications. For example, it is useful 
for the reconstruction of a factory layout or of an robot cell [12]. This rep- 
resentation ranges from point clouds to more complex models, e.g. boundary 
representation (B-Rep) models. In our previous work [3], a method to generate 
planar B-Reps from a single point cloud was presented. Multiple point clouds can 
be incorporated in real-time, providing a valid B-Rep model in each time-step. 
By using the known pose of the camera (e.g. in [3] a robotic manipulator was 
used) two B-Rep models can be merged. The B-Rep is handled by an half-edge 
data-structure. The requirement of a known pose can be lifted by utilizing the 
method from our work [4], by using the angles between three linear independent 
faces to generate a possible rigid body transformation between two B-Reps. 

By combining these approaches, it is possible to generate a B-Rep as represen- 
tation of the world by using a depth sensor, without a known camera pose. This 
enables the use of hand-held cameras, which leads to applications like object 
and robot cell reconstruction. The camera is moved by a user through the scene. 
Every point cloud is reconstructed into a B-Rep and merged into the world 
representation. However, the resulting model is not necessarily complete. One 
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possible reason may be occlusion, as a result of a complex scene. A similar aspect 
are missing points due to absorbed or reflected light. Another problem is that a 
scene can not be reconstructed from too noisy data. A last possible source for 
errors is the user, e.g. when he misses parts of the scene. There are two methods 
for dealing with the problem of incomplete models: On the one hand, sufficient 
small holes inside the representation should be closed automatically to enable a 
comfortable usage of the system. On the other hand, it is necessary to direct the 
user during the recording if the holes can not be closed automatically. 

This paper consists of the following parts: The next section presents the state 
of the art how to handle incomplete representations. We present our approach 
in Section 3. First, we determine attributes to classify holes and afterwards we 
suggest a method to automatically close them and to generate user guidance 
for holes, which can not be closed automatically. The approach is validated in 
Section 4. In the last section, we discuss our contribution and future work. 


2 State of the Art 


The most simplest way to give user guidance is a visualization of the model [5]. 
The model is typically rendered from the user’s view. This visualization helps the 
user detect missing parts. A more advanced method is visual feedback to notify 
the user of missing parts. This can be done by colouring the backsides of the 
model, as they are only visible, when a hole exists and the user can look inside 
the model [1]. Finally, user guidance is possible by suggesting a camera pose, 
from which the hole can be closed. This problem is known as Nezt-Best-View 
in the context of robotics [6,7]. However, a precise pose is unnecessary as it is 
impossible for a human to take exactly this pose. Additionally, in our domain 
of live reconstruction there is no set of possible views, and the human operator 
may have additional knowledge of the scene. Therefore, it is only necessary to 
identify holes and give a hint from which direction to look. It is not necessary to 
give a quality measure for each pose or to propose a reasonable one, as all holes 
should be closed in the end and the selection can be left to the human. Overall it 
is at least necessary to give a visualization to ensure a simple and intuitive live 
reconstruction. This is based on the Hand-Eye-Control System presented in [2], 
which reasons, that a human reacts in such a way, that the difference between 
the current state and a goal state is reduced [1]. 

The other problem mentioned are small holes. For this incompleteness user guid- 
ance can be generated, but it is not always helpful, especially when the holes 
exist because of occlusion or properties of the sensor. Therefore it is necessary to 
automatically close theses holes. This problem is known as Model Repair [8,9]: 
In general, these methods can be split into procedures based on point clouds 
or on meshes. The mentioned methods are typically based on the surface or on 
volumetric grids. In this case we will focus on the mesh-based methods, due to 
the B-Rep as the underlying structure. The surface-based methods follow differ- 
ent approaches. One possible way is to split the hole in smaller, easier to repair 
ones [16]. Another way is based on a triangulation of the hole, with the goal of 


(a) Inner, k = 1 (b) Outer, k = 1 


(c) Inner, k = 3 (d) Outer, k = 3 
Fig. 1: Examples for holes with differ- 
ent orientation and different cardinal- 
ity k. The blue half-edges are the hole, 
the red one the transition-half-edges. 


Fig. 2: used 
variables for an outer hole with 
k = 3. There are two separated 
transition-half-edges and one twin- 
transition-half-edge, the hole-half- 
edges (blue), the transition-half-edges 
(red), transition-vertices (red-dotted), 
and the intersection-point of the three 
face’s planes (green). 
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minimizing the amount of necessary triangles [17]. A last group of methods is 
based on a implicit surface representation and trying to utilize additional knowl- 
edge resp. properties of the object [13-15]. The volumetric approach is based on 
using a volumetric representation, with the idea of expanding the area around 
the hole, and therefore closing the model. These methods are usually more rea- 
sonable for more complex holes. In contrast to these methods, we are using the 
B-Rep data structure, therefore a direct application of theses methods is not 
possible. Methods especially for B-Reps use features like edges and vertices, but 
are also based on a tessellation [10,11]. Additionally, our goal is to use the hole 
closing during the live reconstruction, so there may not be enough information 
to close the hole with the before mentioned methods. 


3 Our Approach 


3.1 Definition and Handling of holes 


Based on B-Rep models, it is easy to determine incomplete parts within a B- 
Rep due to the half-edge data-structure by testing, whether every half-edge has 
a twin-edge. Therefore, a hole is a closed loop of half-edges without twin-edges. 
A hole can be described by its cardinality k, which is the number of adjacent 
faces of the hole. Using this definition, the external boundary of the B-Rep is 
also recognized as a hole. This leads to the definition of the orientation: A hole 
is called inner hole if it can be closed by enlarging every face, which is adjacent 
to the hole. Otherwise, the hole is called outer. Examples for holes with different 
attributes are visible in Figure 1. A last definition are the so called transition- 
half-edges T; (red edges in Figure 1). These are twin-edges which start (or end) 
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in a vertex on the boundary of a hole, but are not part of the boundary itself. 
However, they still have to follow the boundary of the corresponding face. Based 
on this definition, we call the vertices, which are part of the hole and a transition- 
edge starts/ends in them, transition-vertices (red-dotted circles in Figure 1). 
For further formalisms later on, we define the following variables: The involved 
faces f; with i = 1, ..., k of a hole have the normal n; € R. It should be mentioned 
that one physical face may have multiple identifiers following this definition, 
due to the fact, that the cardinality may count one face multiple times. The 
transition-vertex, in which the hole-half-edges switch from face fı to fi; is called 
ti with the position T; € R®. The direction of the transition-half-edge, which 
ends in t; is called h;. Based on this definitions, the vector from a transition- 
vertex t; to the next one tj is called v;; = T; — T;. For the special case of 
k = 3, we call the intersection point of the planes defining the three faces as 
S € R, and the vector from one transition-vertex t; to the intersection point S 
as s; = S — T;. These identifiers are visualized in Figure 2. 

The state of the art indicates two kinds of user support: the automatic closing 
of holes and the user guidance. However, only because a hole can be closed 
automatically, it is not always in the user’s interest to do so. Therefore, we 
distinguish between three kinds of holes a) that can be closed automatically and 
the user agrees, b) that can be closed automatically and the user disagrees, and 
c) that can not be closed automatically. In our previous work [3] and [4] a user- 
defined parameter was introduced, the so called structure size ôs. The structure 
size indicates, which geometric features (considering their size) can be ignored. 
Therefore, all holes smaller than the structure size can be closed automatically, as 
they are too small to be relevant. Otherwise, user guidance should be provided. 
The definition of the size of a hole has to take different shapes of holes into 
account. As the size of a hole the maximum diameter (k = 1), the maximum 
distance to the missing edge (k = 2) or the distance to the missing corner (k = 3) 
can be used. For holes with a cardinality k > 3 or the orientation outer, user 
guidance is the only possibility, as the hole can not be closed automatically. 


3.2 Detection of holes 


The following algorithm detects all holes of a B-Rep, starting with a half-edge 
without a twin-edge. This start edge is added to the set hole. Beginning with 
start, all following half-edges (considering the direction in a B-Rep) are visited 
until we reach the start again. If we encounter an edge with an opposite, we do 
not add this to the set (as this half-edge is a transition-half-edge) and return 
to the hole in the next step. In the end, the set hole is added to the overall 
set holes. This whole process is repeated until no half-edge without twin-edge 
exists, which has not been visited yet. The cardinality is determined by count- 
ing the participating faces. The procedure is summarized in Algorithm 1. The 
function twin(c) denotes the half-edge with the twin relationship to half-edge c. 
Analogously, the function successor(c) denotes the half-edge after c for this hole. 

Based on the now known holes and their cardinality, we calculate the orien- 
tation of holes, starting with k = 2. The idea is, to check whether the transition- 
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Algorithm 1 Determining all holes of a B-Rep 


1: holes = {} 
2: while 3 An unvisited half-edge without twin do 
3: start = random, unvisited, half-edge without twin 


4 mark start as visited 

5 hole = {start} 

6: c = successor(start) 

i while c # start do 

8 if c has a twin then 


9: c = twin(c) > return to the hole 
10: else 

11: hole = hole Uc > add c to the hole 
12: mark c as visited 

13: c = successor(c) 

14: holes = holes U hole > add current hole to the set 


half-edge points in the same direction as the vector from one transition-vertex 
to the next one. However, the transition-half-edges are not necessarily twins. 
Therefore, we take the normals of the corresponding faces into account: 


nı X n2)o vı2 <0 if fi, f2 convex 


outer hole <= f (1) 


(nı x No) Ovig >0 if fı, fo reflexive 
The definition can be easily extended for the case of k = 3: 


(n; x nj)os; <0 if fi, fj convex 
(n;xn;)os;>0 if fi, fj reflexive (2) 
Vi € {1,2,3}, j=(i-1+k) mod k. 


outer hole &> 


This calculation fails, if participating faces are linearly dependent, because no 
intersection point exists. Due to the definition of an inner hole, that faces meet 
in one point, it is clear that a hole with linear depending faces is an outer hole. 
To calculate the orientation we use whether two faces are convex or reflexive. This 
relationship is only defined for edges (as faces can be both on different edges) 
and is calculated by edge conver —> (vij X nj) on; > 0. In our case, we want 
to know the convexity of two faces in one transition-vertex. We define a plane 
T= (nj x nj —(n; x nj) o T) which is orthogonal to both faces and contains 
the transition-vertex t;. We project both faces onto this plane and get two lines 
gi, gj- For these lines there are four possible cases, how the transition-half-edges 
can lie in the plane m. For two of these the decision of the convexity is obvious. 
The other two are undefined, as both convex or reflexive is possible. However, 
an estimation is possible by projecting the hole-half-edges onto 7 and comparing 
the overlap a; resp. a; onto the other side (see Figure 3). Depending on which 
value is greater, the two faces are convex or reflexive in this transition-vertex. 
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(c) undefined (d) undefined 


Fig.3: The four possible cases how 

transition half-edges h; and hj can lie Fig.4: Closing of inner holes (top) by 
in plane 7. The convexity in transition- replacing hole-half-edges (middle) and 
vertex t; (red-dotted) is estimated. merging of separated edges (bottom). 


3.3 Automatic closing of holes and Generation of User Guidance 


Based on the attributes, we close holes automatically which are smaller than 
the structure size 6,. For k = 1 the inner boundary is removed, in which all 
half-edges of the hole lie. For k = 2, all half-edges of the hole are replaced by 
one twin-half-edge. If there are already half-edges on this physical edge, they 
are merged afterwards. For k = 3 a new vertex v, is generated, which lies in the 
intersection of all involved faces. All hole-half-edges are deleted and replaced by 
pairs of half-edges, with v, as one vertex, and each transition-vertex as the other 
vertex. If the edges are separated, they are merged into one (see Figure 4). 
Based on the automatic closing, three combinations of cardinality and orientation 
exist, where an user guidance during the recording is necessary: Inner holes with 
k < 3, outer holes with k < 3, and holes with k > 3. This separation is based 
on the idea, that holes with k < 3 can be closed by taking a look from one 
proper pose. The user guidance aims on the calculation of an arrow consisting 
of a position P € R° and a direction r € R? with ||r|| = 1. 

The idea for the method for inner holes with k < 3 is based on the functionality of 
depth sensors. The easiest way to capture planar surfaces occurs when the angle 
between the face’s normal and the sight ray is as small as possible. Therefore, 
the direction of the arrow can be calculated as the sum of the inverted normals 
of all involved faces. The arrow directs towards a point which is calculated either 
as the centroid P, of the hole or as the weighted sum of the transition-vertices: 


P,-d-r k=1 
r= >, -m/ So nj „P= s(Tı+T3)-d-r k=2 (3) 
i=1..k i=1..k 2 1 


3(Tı de T3)-d-r k=3 
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The parameter d € R corresponds to a distance from the hole in the direction of 
—r which can be adjusted by the scaling of the B-Rep and the user’s preferences. 
For the case of outer holes with k > 3, the missing patches lie on the other side 
compared to inner holes, therefore it is reasonable to use the inverted direction. 
However, when a hole is the inner boundary of a surface, the direction may still 
not fit, because the arrow is on the wrong side of the face. This can be fixed by 
checking, whether the hole is an inner or an outer boundary of this face: 


r= ` a-nj/ So ny 


i=1..k i=1..k 


(4) 


. +1 inner boundary 
with a;= 
j —1 outer boundary 


The closing of holes with k > 3 is ambiguous. The calculation of the orientation 
fails, because no intersection point is available. The idea is to determine triangles, 
which would close the hole and calculate a direction, from which most of them 
are visible. The first step is to calculate the mean M of the transition-vertices as 


M= i >, T;, which can be used as the position. Together with the transition- 
i=1..k 
vertices T;, this point is used to triangulate the hole and calculate the normal 


A;,=(M-T,)x vi; with j = (i + 1) mod k for each triangle. These normals 
are not normalized, so they can be weighted by the area of each triangle: 


r=-) Ay SA 


i=1..k 


(5) 
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4 Validation 


The generation and merging of B-Reps [3,4] was implemented and extended by 
the developed methods. It can be used as an application with common depth sen- 
sors (Microsoft Kinect) to reconstruct objects by moving the hand-held sensor 
through the scene. Also, we implemented an Android application which can be 
used on devices with a depth sensor (Lenovo Phab 2 Pro) as a proof of concept. 
One possible validation technique is the complete exploration of the problem 
space. For this, we limit the cardinality to k < 3 and determine all possible 
cases considering convexity of faces, orientation, and for outer holes whether the 
hole is an outer or inner boundary. For each possible case, a synthetic test case 
is generated and the calculated user guidance evaluated. The decision, whether 
the user guidance is reasonable, is an individual decision. The cardinality is fixed 
with k < 3 due to the fact that the number of possible cases increase exponen- 
tially in k and no automatic closing is possible with higher values for k. Another 
method is a validation based on real data. A first step is to evaluate the auto- 
matic closing of holes. In this case, the reconstruction is done by hand without 
any support. This result can be compared to a reconstruction which uses the 
automatic closing of holes and the same input. A second method is to analyse 
the amount of user guidance on real world data. This experiment is based on 
our previous work [12], in which a robot cell is reconstructed in an offline step. 
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Fig. 5: Inner holes with cardinality k = 3 for all four possible cases considering 
the convexity of the three edges. Top: User guidance. Bottom: The model after 
automatic closing. The direction of the face normal is indicated by a black line. 


This reconstruction was used for a real-time path planning. The developed user 
guidance and hole closing should help in generating a complete model. 

Based on this setup for the validation we obtain the following results: As a proof 
of concept we reconstruct complete objects. The user starts in an arbitrary place 
and surrounds the object. After each merged segment the user guidance is cal- 
culated, and the operator may take another image. After one round the model 
is completed to a high degree, with some indications left (see Figure 6). Consid- 
ering the complete evaluation we generated synthetic models for every possible 
case: If the half edges of the hole lie in more than one inner boundary, the result- 
ing user guidance is not very helpful due to a poor direction. The same problem 
occurs with models in which one face is twisted around another. However, these 
two cases are not highly relevant as exactly this situation with no other faces 
changing the cardinality must arise. Additionally, holes which lie in more than 
one inner boundary are rare in reality. The results for inner holes with k = 3 
and different convexity can be seen in Figure 5. The result of the comparison 
between real world data with and without the automatic closing is presented 
in Figure 7. The difference between these two reconstructions is small, but sev- 
eral holes are closed in a reasonable way. Due to the fact, that a edge can only 
be merged when it exists in both input B-Reps (due to the method from [3]), 
most of the holes exist on edges and corners. If the corresponding faces are well 
reconstructed, the hole is barely visible. The scene in Figure 6 visualizes the 
user guidance. The amount of arrows increases in the beginning and reaches the 
maximum number when the object is circled for the first time. When the user 
actively closes these holes, the number of arrows decreases until the object is 
completely reconstructed and no arrows remain. 
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Fig. 6: Test-scene and the reconstructed model with a Lenovo Phab 2 Pro. During 
the recording, the user guidance was calculated and displayed. 
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Fig. 7: A testscene (c) was reconstructed with a Microsoft Kinect for XBOX. 
Model (a) is the result without automatic closing. In model (b) sufficient small 
holes were closed automatically. In detail (d+e) several holes were closed. 


5 Conclusion 


We presented a new approach to close holes automatically and to generate user 
guidance during recording. We found that different types of holes can exist, 
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which need different methods depending on the attributes of the hole. A method 
to determine all holes of a B-Rep and their attributes was presented. Based 
on this, the holes are closed automatically or user guidance was generated. We 
evaluated our approach with a complete exploration of a problem space and by 
comparing the results of a real world scene with and without automatic closing 
of holes. Future work may include a collision test for the arrows to prevent them 
from being stuck inside the model. Additionally, a user study could verify the 
usefulness of the user guidance and the decrease in recording time. 
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Abstract. The productivity of groups can be increased by enabling 
group members to share their perceptions of the environment. We adapt 
this concept for mobile robots by presenting an object-oriented approach 
to a shared environmental model. The objects are stored in a graph, 
which saves memory and computing power and allows the representation 
of hierarchical and topological relationships. Each object can contain geo- 
metric and semantic data as well as information about its current, past, 
and planned or estimated future movements. An example application 
shows that modeling future motion can prevent collisions. 


Keywords: world models, distributed systems, dynamic objects, col- 
laborative robots 


1 Motivation 


Product customization and shortening of production life cycles increase the de- 
mand for flexible manufacturing systems. Mobile robots show great potential due 
to their high configurability and scalability as well as their freedom of movement. 
The applications are diverse and range from logistics and cleaning or inspect- 
ing to operation at multiple workstations. The large number of mobile resources 
means that future production facilities will require decentralized intelligence to 
handle the increasing complexity. 

In order to navigate and interact safely and efficiently in such facilities, 
robots need an understanding of their environment that is technically enabled 
by environmental models. The combination of the limited perceptions of individ- 
ual robots into a common knowledge base enables collective intelligence. Those 
shared models enable improvements firstly in safety, e.g. warning of imminent 
danger, and secondly in efficiency, e.g. avoiding traffic jams. 

Existing models lack the ability to represent object dynamics. At critical 
points in a production facility, such as intersections and pedestrian crossings, 
information about the dynamics of surrounding objects can improve overall per- 
formance and safety. Dynamic information could either be directly integrated 
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into the map by the performing robot or estimated by observers for agents with 
no connection to the shared model, e.g. human workers. 

This work presents a novel approach for integrating object dynamics into 
world models. The suggested solution consists of a client-server architecture and 
a graph-based data model supporting geometry, semantic classification, topol- 
ogy and object dynamics, including past and present as well as predicted and 
planned future trajectories. First, an overview of environment models with fo- 
cus on papers related to this work is given in section 2. Then, our approach is 
presented in section 3. Section 4 describes a proof-of-concept application offering 
a technical implementation and illustrating the added value of the approach. 
Finally, the results are discussed in section 5 and perspectives for future research 
are given. 


2 Related Work 


Research on shared environmental models focuses on three main areas: methods 
for obtaining new data and integrating them into an environmental model, so- 
lutions for contributing and distributing data in multi-user scenarios and lastly, 
the actual data modeling, which uses data structures, itself. 


Obtaining New Data This area includes the various methods from the fields of 
perception, data fusion and pattern recognition and is considered only briefly 
below. Standard approaches to mapping in mobile robotics are “Simultaneous 
Localization and Mapping” (SLAM) [1, 2] or “HectorSLAM” [3], which gener- 
ate grid-based maps. Alternatively, the raw sensor data can also be processed by 
object recognition and integrated into an object-oriented data model [4]. There 
are several applications in the field of service robotics [5, 6]. Furthermore, data 
can be processed to form topological graphs, e.g. transforming a 2D grid-based 
map of a building into a graph with nodes representing rooms [5, 7]. Appli- 
cations to path planning for multiple mobile robots are given in [8]. Due to 
inaccuracy in perception, objects need to be registered to their absolute position 
in the environmental model, referred to as “anchoring” or “data association” 
[9, 10]. Context-based classification improves anchoring by considering existing 
knowledge [11]. 


Data Distribution This area integrates the methods from the fields of network 
architecture, consistency management and synchronization and, again, is con- 
sidered only briefly below. In contrast to fully centralized map storage and fully 
synchronized maps with all participants, intermediate stages are a trade-off be- 
tween latency and consistency [12]. A client-server structure can be applied to 
an object-oriented distribution model, preserving consistency by predefined ac- 
cess rights to object changes [13]. Multi-server architectures with environmental 
subsets enable scalability [14]. Moreover, there are approaches that are tolerant 
to inconsistency by assuming it to be acceptable at certain points of the data 
model [15]. 
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Data Structures for Environment Mapping Our approach focuses on data 
structures, therefore the related work is discussed in detail. Common approaches 
for geometric relationships can be classified as grid-based or object-oriented. In 
addition to geometric data, environmental models can represent semantics or 
uncertainty. 


Grid-based Structures A fundamental concept of map generation is the “Occu- 
pancy Grid Map” (OGM), discretizing 2D or 3D space into independent cells 
assigned with the estimated occupancy, e.g. “free”, “occupied” or “unknown” 
[1, 3, 5, 16]. The storage effort can be reduced by structuring the grid cells in a 
hierarchical manner, the so-called “octree” [17]. Combinations of several octrees 
enable object hierarchies and multiple resolutions [18]. 


Object-oriented Structures With object trees, not only cubes, but any geometric 
object can be described. Gregor et al. [19] use homogeneous transformations to 
indicate the relative spatial arrangement of parent and child objects. Seman- 
tic information can be represented by linking the nodes of the object tree to 
corresponding elements of a “conceptual hierarchy” [5]. 

An extension to graph structures holding geometric and transformation nodes 
is presented by Blumenthal et al. [20]. Here, nodes can contain semantic informa- 
tion via key-value pairs and past object movements are traced by transformation- 
timestamp pairs. The graph structure allows several transformation nodes to 
point to the same geometry node, which enables the modeling of competing 
perceptions in multi-robot scenarios. Multi-user operation is considered unprob- 
lematic because geometric objects, as well as transformation-timestamp pairs, 
are invariant. In addition, there is an interface for distributed storage of a graph, 
but possible consistency problems are not considered. Other approaches extend 
geometric models by separately stored ontologies [10, 21], which are not in the 
scope of this work. 

Further graph structures, such as in [22], focus on the shared use of the 
map by several actors. For this purpose, a so-called “generative grammar” is 
set up, which regulates how parts of the model may change. The emphasis of 
the research can also be on a systematic class-based representation of geometric 
information [23]. 


Modelling of Uncertainty Predictions or measurements result in uncertain data, 
which must be correctly modeled. By augmenting coordinates with one standard 
deviation per dimension, locations can be stored as Gaussian distributions [24]. 
Likewise, uncertain occupancies in octrees can be expressed by probability val- 
ues [17] and spatial transformations can be extended by additional covariance 
matrices, expressing 6D spatial deviation [20]. 

Different approaches introduce confidence values for graph attributes and 
links [10]. Object classification algorithms provide a classification probability, 
which is integrated into the class structure by Papp et al. [23]. Additionally, 
generic uncertainty models can be linked to physical quantities here. 
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3 Scientific Approach 


The overview of the scientific approaches to environmental modeling shows that 
so far there are only solutions that are specialized in one field. If several robots 
with a shared environmental model are used, their activities and perceptions 
are very homogeneous. This in turn leads to restrictions in the representation 
of the perceived world in the model. Comprehensive environmental models, on 
the other hand, are not prepared for multi-user or distributed systems. It is not 
possible to enter predicted trajectories of robots or dynamic obstacles into the 
world model in any of the considered solutions. 


3.1 Requirements 


From the shortcomings of the existing solutions mentioned in the previous section 
and the generally necessary characteristics of an environmental model, we can 
derive some requirements for a new, comprehensive system for mapping the 
environment. 


Geometry and Dynamics Real world objects should be depicted in terms of their 
position and shape. In order to adequately reproduce movements of objects, a 
distinction must be made between past, present and future movements. The 
latter must be further subdivided into planned and estimated movements. 


Combination of Topological, Geometrical and Hierarchical Relations Three 
types of relations are defined in the literature: 


— hierarchical, indicating a dependency of the function and location of subor- 
dinate objects on their parent, 

— geometrical, indicating a spatial relation, and 

— topological, indicating neighborhoods that are not necessarily directly con- 
nected to the spatial location. 


For compact storage, a single data structure should represent these three types 
of relations. 


3.2 Scientific Concept 


Our system is based on a client-server architecture similar to those in [13] and 
[14]. Each instance contains a copy of the shared environmental data. In the 
following, we describe how the data is structured and represented, focusing on 
motion modeling. 
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Structure of the Data Model Our approach is based on object-oriented 
representation. Geometric or abstract objects of the real world are mapped to 
nodes of a data structure whose edges reflect relationships between the objects. 

The data structure has two types of object links, vertical and horizontal. The 
relative position of two objects is stored in vertical links. They form a tree-like 
structure and therefore also show hierarchies between objects. In contrast to a 
conventional tree, objects can have more than one parent node in the superordi- 
nate level, resulting in a directed acyclic graph (DAG). This means that multiple 
memberships can be mapped, for example a workpiece that is jointly transported 
by two robots. If an object has more than one parent node, its absolute position 
is ambiguous because there are several paths to the root node. 

The second type of object connection is the horizontal link. It shows the 
topological neighborhood of two objects. Horizontal links may interconnect any 
objects in the DAG of vertical links. The collection of all horizontal links in a 
data model forms one or more graphs. 


Contents of the Data Model Each node of the data model represents a 
real physical object or a group of several subordinate objects [18, 20]. Thus 
a node does not necessarily have to contain a geometry. The assignment of a 
classification and a trajectory is also possible so that all use cases can be covered. 


Geometry and Classification In our approach, several types of geometry repre- 
sentation can be stored. Point clouds provide a storage option close to the sensor, 
while octrees provide a storage-efficient solution. Moreover, triangle meshes can 
be stored, which are very suitable for the visualization of objects. The techni- 
cal details are not discussed in detail here; instead, readers are referred to the 
literature [17, 20]. 

In addition to geometries, the results of classification algorithms can be stored 
in the nodes [10, 23]. 
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Fig.1. (a, b) Dispersing trajectories of velocity and orientation (yaw) angle plotted as 
mean value and standard deviation (dashed) [25] (c) Discrete trajectory of uncertain 
positions [25] (d) Test-setup of first robot (top), human worker (middle) and observing 
robot (right) 
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Object Dynamics Present and past object dynamics are modeled according to 
[20]. This is applicable to the linear and angular velocity stored in each object 
node, as well as to the location of objects stored in the vertical links. 

For interfering objects, future motions can be predicted based on present 
and past motions. Our approach includes an uncertainty model for representing 
predicted trajectories [25]. Figures 1(a) and 1(b) show dispersing trajectories of 
velocity and orientation angle, expressed as the mean value and standard devia- 
tion. Accordingly, discrete trajectories are formed by time-sampled points where 
the position is assigned an uncertainty in the form of a covariance matrix, see 
Figure l(c). This representation of an uncertain trajectory has the advantage 
that time and location components are separated. In an environmental model 
which is based on geometric relationships, the trajectory can therefore be seam- 
lessly inserted. 

In case of robots, future motions are predictable. Therefore, our approach in- 
cludes planned trajectories as continuous paths (NURBS: Non-Uniform Rational 
B-Spline) mapped to time. 


4 Proof of Concept 


A typical system has been created using open-source software. Due to the early 
stage of development, an industry-like application has been simulated in an office 
environment, e.g. an order picking robot in a small parts warehouse. 


4.1 Implementation 


The described object-oriented data model has been written in C++. Several 
hierarchical classes represent the different nodes and edges as well as the optional 
information about classification, geometry and trajectory. They are depicted 
on the left side of Figure 2. Linux-based client programs were implemented for 
reading sample geometries from a database, inserting a trajectory and visualizing 
the model. Another ROS-based client makes the world model accessible to the 
mobile robot. 

The clients communicate via a WLAN with a linux-based server application. 
If a client makes a change to its instance of the data model, it informs the server. 
The server forwards the change to the other clients. 


4.2 Experiment and Observation 


The use example consists of a mobile robot driving along a narrow alley. At a 
hard-to-observe intersection a human worker crosses the robot’s pathway. 

The robot, which is only equipped with on-board sensors, would need to 
reduce its speed at such locations to ensure safety. Therefore the robot’s perfor- 
mance decreases even if no interference occurs. 

In this scenario it is assumed that a second robot beyond the junction can 
perceive the person and estimate their future movement. The expected trajectory 
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Fig. 2. Classes of the implemented data model (left), typical view of the experiment 
(middle) and matching environment model including predicted trajectory as circles on 
the floor (right) 


is fed into the environmental model available to the first robot. The setup is 
illustrated in Figure 1 (d). 

Experiments prove that the first robot can react to the interference by stop- 
ping before being able to perceive the human worker based on onboard sensors 
alone. Figure 2 shows the crossing situation in reality and in the model. 


5 Results, Discussion and Conclusion 


In this work, a novel shared environmental model for mobile robots was pre- 
sented. The graph-based data model can represent geometry as point clouds or 
triangle meshes and semantic information as classification results. The proposed 
solution is suitable to represent future or past object dynamics and to consider 
the uncertainty in predicted trajectories, as demonstrated in the section above. 

Furthermore, the environmental model offers advantages in the operation 
of mobile robots. By communicating with other robots, a robot can react to a 
moving obstacle before it has detected it with it’s on-board sensors. 

Robots must normally drive at an appropriate speed to be able to brake at 
dangerous points such as hard-to-observe intersections. It is also conceivable to 
install stationary sensors at danger points so that potentially interfering objects 
are always reported to the shared environmental model. Thus overall perfor- 
mance can be increased while maintaining safety. 


5.1 Fulfillment of Requirements 


Current movements can be stored in the model as single values with a timestamp. 
When a value with a newer timestamp is added, the previous one moves to 
the history. Future movements can be stored continuously or discretely with 
uncertainty. In this way, the degree of detail and the time horizon can be varied 
arbitrarily. This covers a large number of applications. 

Combining hierarchies and location relationships makes the modeling of lo- 
cation-independent hierarchies more difficult. On the other hand, it simplifies 
changing the location of object groups as only the position of the parent node 
must be updated. 
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5.2 Outlook 


Further Aspects of Modeling for Multiple Robots In addition to the aspects de- 
scribed above, the presented concept contains mechanisms to ensure consistency 
and efficient distribution of data. These are of great importance in a shared 
environmental model. The mechanisms are based on the existence of a central, 
superordinate server that decides on the approval and distribution of updates. 
To limit network traffic, it is possible to subdivide the environmental model into 
several regions. In this way, each robot receives updates matched to its location 
instead of receiving updates for the whole map. The horizontal links between 
the model objects that are described above are decisive for the subdivision. 


Future Work Several possibilities for geometry storage were mentioned in sec. 3.2. 
It should be investigated whether further methods need to be added to this se- 
lection. Since most types of information occurring in a virtual environment are 
uncertain, the modeling of uncertainty should also be further investigated. Fi- 
nally, tests of the environmental model with large amounts of content must be 
carried out to assess storage efficiency and speed so that the results can be 
compared with other solutions. 


6 Acknowledgements 


The results presented in this article were developed within the FORobotics (AZ- 
1225-16) research network. The authors would like to thank the Bavarian Re- 
search Foundation and all participating project partners for their funding and 
support of the project. 


References 


1. Chatila, R. & Laumond, J. Position referencing and consistent world mod- 
eling for mobile robots in 1985 IEEE International Conference on Robotics 
and Automation Proceedings 1985 IEEE International Conference on Robotics 
and Automation Proceedings. 2 (Mar. 1985), 138-145. 

2. Durrant-Whyte, H. & Bailey, T. Simultaneous localization and mapping: 
part I. IEEE Robotics Automation Magazine 13, 99-110 (June 2006). 

3. Kohlbrecher, S., Stryk, O. v., Meyer, J. & Klingauf, U. A flexible and 
scalable SLAM system with full 3D motion estimation in 2011 IEEE Inter- 
national Symposium on Safety, Security, and Rescue Robotics 2011 IEEE 
International Symposium on Safety, Security, and Rescue Robotics (Nov. 
2011), 155-160. 

4. Mason, J. & Marthi, B. An object-based semantic world model for long-term 
change detection and semantic querying in 2012 IEEE/RSJ International 
Conference on Intelligent Robots and Systems 2012 IEEE/RSJ Interna- 
tional Conference on Intelligent Robots and Systems (Oct. 2012), 3851- 
3858. 


10. 


11. 


12. 


13. 


14. 


15. 


16. 


53 


Galindo, C. et al. Multi-hierarchical semantic maps for mobile robotics in 
2005 IEEE/RSJ International Conference on Intelligent Robots and Sys- 
tems 2005 IEEE/RSJ International Conference on Intelligent Robots and 
Systems (Aug. 2005), 2278-2283. 

Marton, Z.-C., Pangercic, D., Blodow, N. & Beetz, M. Combined 2D-3D 
categorization and classification for multimodal perception systems. The 
International Journal of Robotics Research 30, 1378-1402 (Sept. 1, 2011). 
Goerke, N. & Braun, S. Building semantic annotated maps by mobile robots 
in Proceedings of the conference towards autonomous robotic systems (2009), 
149-156. 

Binder, B. Spatio-temporal prioritized planning Diplomarbeit (Technische 
Universität Wien, 2017). 

Coradeschi, S. & Saffiotti, A. Anchoring symbols to sensor data: preliminary 
report in Proceedings of the Seventeenth National Conference on Artificial 
Intelligence and Twelfth Conference on Innovative Applications of Artificial 
Intelligence (2000), 129-135. 

Baum, M., Gheta, I., Belkin, A., Beyerer, J. & Hanebeck, U. D. Data asso- 
ciation in a world model for autonomous systems in 2010 IEEE Conference 
on Multisensor Fusion and Integration 2010 IEEE Conference on Multisen- 
sor Fusion and Integration (Sept. 2010), 187-192. 

Günther, M., Ruiz-Sarmiento, J. R., Galindo, C., González-Jiménez, J. 
& Hertzberg, J. Context-aware 3D object anchoring for mobile robots. 
Robotics and Autonomous Systems 110, 12-32. ISSN: 0921-8890 (Dec. 1, 
2018). 

Fleury, C., Duval, T., Gouranton, V. & Arnaldi, B. A new adaptive data 
distribution model for consistency maintenance in collaborative virtual en- 
vironments in. Joint Virtual Reality Conf. of EGVE 2010 - The 16th Eu- 
rographics Symposium on Virtual Environments, EuroVR 2010 - The 7th 
EuroVR (INTUITION) Conf., VEC 2010 - The Annual Virtual Efficiency 
Congress (2010), 29-36. 

Singh, G., Serra, L., Png, W., Wong, A. & Ng, H. BrickNet: sharing object 
behaviors on the Net in Proceedings Virtual Reality Annual International 
Symposium ’95 Proceedings Virtual Reality Annual International Sympo- 
sium ’95 (Mar. 1995), 19-25. 

Hesina, G., Schmalstieg, D., Furhmann, A. & Purgathofer, W. Distributed 
open inventor: A practical approach to distributed 3D graphics in Proceed- 
ings of the ACM symposium on Virtual reality software and technology 
(1999), 74-81. 

Kharitonov, V. Y. A Consistency Model for Distributed Virtual Reality Sys- 
tems in 2009 Fourth International Conference on Dependability of Com- 
puter Systems 2009 Fourth International Conference on Dependability of 
Computer Systems (June 2009), 271-278. 

Shaik, N., Liebig, T., Kirsch, C. & Müller, H. in KI 2017: Advances in 
Artificial Intelligence (eds Kern-Isberner, G., Fürnkranz, J. & Thimm, M.) 


54 


TT: 


18. 


19. 


20. 


21. 


22. 


23. 


24. 


25. 


Lecture Notes in Computer Science 10505, 249-261 (Springer International 
Publishing, Cham, 2017). ISBN: 978-3-319-67189-5. 

Wurm, K. M., Hornung, A., Bennewitz, M., Stachniss, C. & Burgard, W. 
OctoMap: A probabilistic, flexible, and compact 8D map representation for 
robotic systems in Proc. of the ICRA 2010 workshop on best practice in 3D 
perception and modeling for mobile manipulation 2 (2010). 

Wurm, K. M. et al. Hierarchies of octrees for efficient 3D mapping in 
2011 IEEE/RSJ International Conference on Intelligent Robots and Sys- 
tems 2011 IEEE/RSJ International Conference on Intelligent Robots and 
Systems (Sept. 2011), 4249-4255. 

Gregor, R., Lutzeler, M., Pellkofer, M., Siedersberger, K.-H. & Dickmanns, 
E. D. EMS-Vision: a perceptual system for autonomous vehicles. [EEE 
Transactions on Intelligent Transportation Systems 3, 48-59 (Mar. 2002). 
Blumenthal, S., Bruyninckx, H., Nowak, W. & Prassler, E. A scene graph 
based shared 8D world model for robotic applications in. Proceedings - IEEE 
International Conference on Robotics and Automation (2013), 453-460. 
Ulbrich, S., Nothdurft, T., Maurer, M. & Hecker, P. Graph-based context 
representation, environment modeling and information aggregation for au- 
tomated driving in 2014 IEEE Intelligent Vehicles Symposium Proceedings 
2014 IEEE Intelligent Vehicles Symposium Proceedings (June 2014), 541- 
547. 

Manso, L. J. et al. Deep Representations for Collaborative Robotics in 
Brain-Inspired Computing (eds Amunts, K., Grandinetti, L., Lippert, T. 
& Petkov, N.) (Springer International Publishing, 2016), 179-193. ISBN: 
978-3-319-50862-7. 

Papp, Z., Brown, C. & Bartels, C. World modeling for cooperative intel- 
ligent vehicles in 2008 IEEE Intelligent Vehicles Symposium 2008 IEEE 
Intelligent Vehicles Symposium (June 2008), 1050-1055. 

Roth, M., Vail, D. & Veloso, M. A real-time world model for multi-robot 
teams with high-latency communication in Proceedings 2003 IEEE/RSJ 
International Conference on Intelligent Robots and Systems (IROS 2003) 
(Cat. No.03CH37458) Proceedings 2003 IEEE/RSJ International Confer- 
ence on Intelligent Robots and Systems (IROS 2003) (Cat. No.03CH37453). 
3 (Oct. 2003), 2494-2499. 

Wiest, J., Höffken, M., Kreßel, U. & Dietmayer, K. Probabilistic trajectory 
prediction with Gaussian mixture models in 2012 IEEE Intelligent Vehicles 
Symposium 2012 IEEE Intelligent Vehicles Symposium (June 2012), 141- 
146. 


55 


Open Access This chapter is licensed under the terms of the Creative Commons Attri- 
bution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which 
permits use, sharing, adaptation, distribution and reproduction in any medium or for- 
mat, as long as you give appropriate credit to the original author(s) and the source, 
provide a link to the Creative Commons license and indicate if changes were made. 

The images or other third party material in this chapter are included in the chap- 
ter’s Creative Commons license, unless indicated otherwise in a credit line to the 
material. If material is not included in the chapter’s Creative Commons license and 
your intended use is not permitted by statutory regulation or exceeds the permitted 
use, you will need to obtain permission directly from the copyright holder. 


Octree-based, multiresolution environment 
mapping for efficient calculation of collision-free 
paths for industrial robots 


Benjamin Kaiser, Veronika Breidohr, and Alexandxer Verl 


Institute for Control Engineering of Machine Tools and Manufacturing Units (ISW), 
University of Stuttgart, Seidenstraße 36., 70174 Stuttgart, Germany 
benjamin.kaiser@isw.uni-stuttgart.de, 


Abstract. For the automatic calculation of collision-free robot paths, 
an environment model is required for collision avoidance. The accuracy 
requirements for the environment model depend very much on the spe- 
cific task of the robot. The computation time of the collision calculation 
increases strongly with higher resolution and dimensions of the environ- 
ment model. For this reason, the demand for high accuracy and fast 
calculation of the collision-free path is contradictory. To solve this con- 
tradiction, we propose a method based on the multi-resolution property 
of octree based maps. Using a subdivision of the workspace into different 
subareas and a set of poses to be connected by path planning, a multi- 
resolution map is created. The individual resolution levels depend on the 
accuracy requirements of the subareas which are defined by the process 
requirements. The presented method is evaluated using the ”KaBa” path 
planning framework. For the evaluation, point clouds scanned with a line 
laser are tested in two path planning scenarios. 


Keywords: path planning, multi-resolution map, octomap, industrial 
robot, collision avoidance 


1 Introduction 


In production technology, the trend towards individualization of products is 
playing an increasingly important role. While the complexity of manufactured 
products is increasing, product life cycles are becoming shorter and shorter. 
In the case of highly customizable parts, a variant may only be manufactured 
once over the entire life cycle of the product. Meeting the requirements down 
to lot size one poses great challenges for production technology. This applies 
in particular to the cost-effectiveness of manufacturing. Innovative manufactur- 
ing systems with a high degree of automation and flexibility are necessary to 
enable the manufacturing industry to meet the new challenges of the changing 
market. Industrial robots offer this necessary flexibility to produce small series 
efficiently. Constantly decreasing acquisition and operating costs also support 
their use. Nevertheless, industrial robots have so far been used mainly in the 
automated production of large series such as in the automotive industry and are 
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rarely found in small series production. The main reason for this is that the ex- 
isting difficulties in preparing robot tasks prevent their economical use in small 
series manufacturing. The programming of industrial robots is a difficult task. 
Despite the use of the CAD/CAM chain, in practice several iterations of robot 
programming are necessary. This means that the programming time is often 
longer than the actual production time. This is especially true for small series. 
In addition, an individual program must be created for each variant. To enable 
the economic use of industrial robots in small series manufacturing, downtimes 
resulting from the preparation of tasks such as programming must be reduced 
or avoided. This can be achieved by automatic path planning for industrial 
robots. In this case, compliance with the process requirements and avoidance of 
collisions of the robot with the environment or with itself must be ensured. A 
complete model of the environment and the robot is required for the planning. 
Path planning and collision detection should be performed with high accuracy 
and as fast as possible. However, these two requirements are contradictory. For 
an exact path calculation, the space must be explored very precisely. At each 
explored point in space, the inverse kinematics must first be solved to check 
whether the point can be reached kinematically. For all reachable points, it has 
to be checked if there are no collisions of the robot with itself or its environment. 
The calculation of collisions is computationally intensive and depends mainly on 
the accuracy of the collision models of the robot and its environment. To resolve 
the contradiction between fast and accurate path calculation, the use of efficient 
maps as environment models plays a particularly decisive role. Therefore, this 
paper presents a novel environment model that allows fast and accurate path 
planning by using different resolutions within the same map. 


2 State of the art 


Models for 3D environment mapping represent a key role in the automatic path 
planning of robots. Especially in the field of mobile robotics this topic is inten- 
sively investigated. With recent advances of automated path planning for in- 
dustrial applications like welding or assembly, environment modeling gets more 
important in the field of industrial robotics as well. Early work mainly focused 
on the registration of height maps by distance sensors [5]. In this early works, 
maps were composed of a fixed, two dimensional grid with a height value at each 
grid point. Multi-height maps followed up on this approach. These can store 
more than one height value per field and can therefore also display passages [8]. 
However, since industrial robots move in three-dimensional space and have 6- 
DOF for approaching poses, this type of map is not suitable for path planning of 
industrial robots. First approaches for 3D environmental models were based on a 
fixed grid whose cells represented voxels [6]. However, due to the fixed grid, each 
field must be stored and initialized whether it is occupied or not. This makes the 
memory requirements for larger maps very high and hence the approach unsuit- 
able for many applications. In addition to grid-based approaches, the use of point 
data from distance sensors such as LIDAR or depth cameras is widespread. Here 
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the map information is stored as point cloud data [1]. The work of [2] deals with 
multi-resolution maps based on point clouds. The disadvantage of using point 
clouds is that they do not provide a volumetric representation of the environ- 
ment. This makes collision detection more complicated since collisions between 
points can remain undetected. Furthermore, these maps are very susceptible to 
noise from the used sensor [7]. Using octrees as a data structure to store maps 
solves the main problem of grid-based maps [9]. By using the octree, the entire 
map does not have to be initialized. Nodes are gradually inserted into the map. 
The use of octrees is very memory efficient. Initially Octree based maps only 
modeled the occupancy of space [3] [10]. The work of [13] and [7]presents a novel 
probabilistic octree based map called Octomap. The Octomap 3D occupancy 
mapping framework models occupied, free and unknown space. The Octomap 
is used as an environment map in a framework for camera-based path planning 
for industrial robots (KaBa) [12]. The path planning framework automatically 
calculates collision-free paths for industrial robots. The calculated paths can be 
executed directly on a robot controller. The KaBa framework first captures a 
3D point cloud of the environment and generates an Octomap. The point cloud 
is used to localize components. This allows the path planning to be adapted 
to deviations of the component from the target geometry and deviations of the 
orientation and position. The KaBa framework has the objective of making the 
use of industrial robots in small series production cost-efficient. 


3 Methodology 


3.1 Problem Formulation 


For a given path planning request consisting of a discrete sequence of n poses 
P;,i € [0,n — 1], a collision-free path must be determined. The path planning 
calculates a smooth, collision-free path connecting these poses. For each segment 
Si—1, between P;_ı and Pi, a feasible set is defined. The path must be contained 
within this subset of the space. The requested path and maximum tolerance 
are process dependent. A welding or assembly process may require very small 
tolerances. Whereas the tolerance for a handling task, where the only goal may 
be the collision-free path, can be selected accordingly large. The computation 
time of automatic path planning depends mainly on two factors. First, the step 
size of the path planning has a strong impact on the calculation time [4]. For 
each pose explored in space, the inverse kinematics of the manipulator must be 
solved. If the pose can be reached kinematically, collisions must be discarded [12]. 
Second, the resolution of the environmental model. This has a direct influence on 
the number of necessary collision checks at each location. Therefore, a reduction 
of the map resolution directly leads to shorter planning times. However, it also 
reduces the accuracy of path planning. This may violate the process requirements 
regarding precision. In order to overcome the contradiction between accuracy 
and computation efficiency, the environmental model has to be reduced without 
having a negative impact on planning accuracy. 
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Based on the segments S;_ı,; and the corresponding tolerances 6;, the area 
in which the tool center point moves along the planned path is limited. The 
process takes place in this area. A reduction of the resolution has a direct effect 
on the process execution. Therefore, the resolution of the environmental model 
cannot be reduced in this area. Nevertheless, collisions must be checked with the 
remaining areas of the map. It must be ensured that the robot does not collide 
with one of its elements during path execution. By splitting the map into sev- 
eral subareas, the overall map size can be reduced without affecting the required 
accuracy along the path. This reduction reduces the planning time. The environ- 
ment model, divided into several subareas, is called Multimap. The Multimap 
is built on octree based maps. There are three reasons for choosing octrees as 
a basis. First, maps based on an octree as an internal data structure contain 
several resolutions inherently. This property can be used directly to generate 
the Multimap. Secondly, an octree represents voxels. A voxel models a volume 
other than a point cloud. If the resolution of a point cloud is greatly reduced, 
the distances between the points may become so large that collisions between 
them cannot be detected. This is not the case with voxels. Third, operations on 
the octree can be performed very efficiently [11]. 


3.2 Resolution areas 


Subareas in Multimap can be parameterized and are dependent on the desired 
path and the corresponding process tolerances. Resolution subareas are defined 
as cylinders with hemispheres at the ends. The subareas are aligned to the path 
segments S;_1;. Each subarea has a defined voxel size. The figure 1 shows the 
division of the map into subareas. The parts of the map that are not within any 
subarea are assigned the parameterized minimum resolution. 


Fig. 1: Subarea division 


The maximum and minimum resolution, as well as the number of resolu- 
tion subareas, can be configured. Since the exact voxel sizes are already defined 
through the underlying octree it is not possible to create the Multimap with 
arbitrarily sized voxels. Therefore, the desired highest and lowest resolution is 
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mapped to the nearest, higher resolution contained in the octree. The resolu- 
tions of the areas between the minimum and maximum level are interpolated 
and mapped to an octree depth as well. 

The parameterization of the multimap is done by the user based on the task 
and the setup of the robot. The minimum voxel size is determined based on the 
process requirements and tolerances. Also, the physical dimensions of the robot 
and especially the end effector must be considered. If the voxels are chosen too 
large, a path may not be possible due to collisions with the environmental model. 
The size of the end effector must also be taken into account when defining the 
size of the areas. For the number and size of the areas as well as the maximum 
voxel size, the calculation efficiency and the resulting movement restriction for 
the robot must be considered. 


3.3 Multimap Generation 


Each path planning request contains an existing octree as an environment model. 
For each request, a multimap based on this octree is generated in a pre-processing 
step. This step consists of two parts. In the first step, the resolution areas around 
each segment of the requested path are created accordingly to the parametriza- 
tion. Cylinders are used to specify the different areas because whether a point 
is contained by a cylinder can be checked very efficiently. However, any set that 
provides a method to check if a point is contained or not can be used. The calcu- 
lated resolution areas specify the maximum depth of the octree for each location. 
The algorithm 1 shows the procedure for determining which nodes of the octree 
are included in the multimap. 


input : A octree and subareas 
output: A list of nodes that are included in the Multimap 
1 for node + first node to last node do 
2 the root (first node) is in the multiMap by default; 
3 if node is in the multiMap then 
4 if depth of node == depth of one of the resolutionAreas then 
5 area := the one of the resolutionAreas at this depth ; 
6 if node is inside area then 
7 | mark node with true; // node and children are added 
8 else 
9 | mark node with false; // node is added 
10 end 
11 else 
12 mark node with true; 
13 end 
14 end 
15 end 


Algorithm 1: Node selection algorithm 
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First, the algorithm iterates over the complete tree and checks for each node 
whether it is part of the Multimap or not. All children of a node that is outside 
the resolution areas are not added to the Multimap. Only the nodes in the depths 
that correspond to a resolution area have to be checked. The others are set to 
the minimum parametrized resolution. Since a lot of nodes are ruled out early 
by not being contained in any resolution area, the determination is fast, even as 
the number of nodes expands drastically with every depth. The nodes that are 
part of the multimap are stored in a list. 


1 Function addToTree (octoMapNode, multiMapNode) input: one node of 
octoMap and multiMap each 


2 if octoMapNode is marked and octoMapNode has children then 

3 for i+ 0 to 7 do 

4 if child i of octoMapNode exists then 

5 add child 7 to multiMapNode; 

6 call addToTree (child i of octoMapNode, child i of 
multiMapNode); 


7 end 
8 end 
9 end 


Algorithm 2: Map building algorithm 


In the next step, the Multimap is created by building a new octree. The 
procedure to build the Multimap based on the node list from the first step is 
shown in algorithm 2. This algorithm constructs a new octree containing the 
previously identified nodes. 


4 Experiments 


Since the Multimap is generated for each path planning request based on the 
already existing octree this step leads to a calculation overhead in comparison to 
using the existing map. The idea behind the Multimap is to reduce the overall 
number of voxels per map and hence speed up the path planning due to the 
reduced amount of collision queries needed. 

This section evaluates the effectiveness of the Multimap. For this purpose, two 
path planning requests are planned and evaluated using the path planning frame- 
work ”KaBa” from [12]. The framework for camera-based path planning scans 
the environment and generates an Octomap from the registered point cloud. The 
framework includes a multi-level collision detection based on bounding volume 
hierarchy and approximate convex decomposition of the robot geometries for 
use with the GJK algorithm. Collisions along the requested path are avoided by 
using algorithms like RRT or A* without violating the defined path tolerances. 


(a) Octomap (b) Test case 1 (c) Test case 2 


Fig. 2: Pathplanning with octoMap and multiMap 


The used Octomap contains a box on a pallet. The map was created from a 
point cloud scanned with a line laser. The first test case is a path along one edge 
of the box. In the second test case, the robot moves the tool into the box. For this 
experiment, file-based Octomaps are used. The path planning is done in Matlab. 
The Octomap import and the generation of the Multimap are done in C++. The 
goal of the Multimap is to speed up the path planning process. To achieve this, 
the time-saving in collision detection has to be big enough to compensate for the 
computation time of the Multimap. If it has almost the amount of nodes that 
the Octomap has, it cannot speed up the collision detection, and additionally, 
take a long time to build. In contrast, the Multimap is especially useful if the 
important areas are small. 
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a) Octomap b) Multimap 1-1 c) Multimap 2-1 d) Multimap 3-1 


Fig. 3: Test case 1 


Three different parametrizations of Multimap are tested. Multimap 1 has 
three resolution areas, with the radiuses 70mm,140mm, and 250mm. The mini- 
mum requested voxel size is 40 mm. The resolution resulting from that request 
is 25.6mm. The maximum allowed voxel size is 300mm. This leads to 204.8mm 
sized voxels. Multimap 2 uses the same minimal and maximal voxel sizes, but 
only one resolution area with radius 140. Multimap 3 uses two resolution ar- 
eas with radii 100 and 250, but with the requested maximum voxel size of only 
150, resulting in the voxel size 102.4mm. The resulting Multimaps of test case 
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1 are shown in figure 3. The Voxels are colored according to their size. Three 
resolution areas result in four different voxel sizes. The three voxel sizes in the 
corresponding areas, and the voxel size outside of any subarea. The resulting 


maps for test case 2 are shown in figure 4. 


a) Octomap b) Multimap 1-2 c) Multimap 2-2 


Fig. 4: Test case 2 


4.1 Reduction of voxels 


SLL 


d) Multimap 3-2 


The reduction of the voxels by the multimap depends on the parameters. This 
reduction increases with smaller resolution areas. Table 1 shows the number of 
voxels in each layer of the Octomap and the generated Multimaps. Which layers 
are compressed depends on the parameterization of the Multimaps. Multimap 
reduces the number of voxels in the required resolution (depth 7) between 83% 
and 97,5% compared to Octomap. In the Octomap all resolutions are included 
and take up memory space, although they are never used. In the Multimap, the 


resolutions that are finer than necessary are cut. 


1234 5 6 7 8 9 10 
Octomap 4 8 20 55 196 701 2901 11442 42465 76848 
Multimap 1-1/4 8 20 55 57 68 115 0 0 0 
Multimap 2-1/4 8 2055 24 90 425 0 0 0 
Multimap 3-1/4 8 20 55 196 132 147 0 0 0 
Multimap 1-2/4 8 2055 99 80 72 0 0 0 
Multimap 2-2/4 820 55 29 100 479 0 0 0 
Multimap 3-2/4 8 20 55 196 218 106 0 0 0 


Table 1: Octree nodes per layer in Octomap and Multimap 


4.2 Planning times 


Table 2 shows the resulting planning times for the two test cases. All times 
are the averages of six runs. The first column of 2 shows the time to import 
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the Octomap. This includes reading the binary file, creating the multimap and 
converting it into a Matlab-compatible format. The import of the Octomap 
takes the most time, therefore the time difference between the import of the 
Octomap and the Multimap is only between 0.05481s and 0.0954s. The times for 
the path planning are different. Multimap times are usually shorter. In test case 
1 Octomap is partially faster. The reason for this is that there are no collisions. 
Altogether the collision calculation takes less time. Since no collisions have to be 
avoided here, space is only explored along the requested path. Nevertheless, the 
total planning time, which also includes times for path preprocessing and other 
additional times, is always shorter with Multimap. Multimap shows an advantage 
in memory consumption and since its tree is pruned, it can be traversed faster. 
The use of Multimap accelerates the overall process. 


preprocessing|path planning |total time 
Octomap -1 0,23059s 1,2367s 3,1995s 
Multimap 1-1) 0,29217s 1,40057s 2,38873s 
Multimap 2-1) 0,30200s 1,19960s 2,18203s 
Multimap 3-1| 0,28667s 1,72902s 2,90948s 
Octomap -2 0,22969s 1,92518s 6,16085s 
Multimap 1-2} 0,29150s 1,44633s 3,86993s 
Multimap 2-2) 0,32517s 1,44283s 3,90612s 
Multimap 3-2 0,2845s 1,25317s 3,56523s 


Table 2: pathplanning times of Octomap and Multimap in comparison 


5 Conclusion 


This paper presents a method for generating 3D maps for the automatic path 
planning of industrial robots. The Multimap uses octrees as a basis and allows 
the direct use of multiple resolution levels of the octree for path planning. The 
division of the map into subareas of different resolution offers the possibility to 
solve the contradiction between exact and fast path planning. The first exper- 
iments show that the Multimap leads to overall faster planning times despite 
the calculation overhead. In further steps, the Multimap needs to be examined 
further using the example of a more complex path planning scenarios with many 
collisions and space exploration where the benefit of Multimap can be seen more 
clearly. 

Due to the promising first results, the multimap should be integrated into 
the native implementation of the KaBa path planning framework for subsequent 
work. The change from the Matlab implementation to the native variant of KaBa 
allows much shorter planning times and allows experimental validation on a real 
robot cell. Besides, the question of how the parameterization of the multimap 
can be carried out in practice by the user must be further investigated. 
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Abstract. The applicability of exoskeletons in different daily and occupational 
settings is continually increasing. Due to varying tasks and user groups, the adapt- 
ability of exoskeletons is becoming more significant, with increasing demands 
for smarter devices. The implementation of force sensors in exoskeletal interfaces 
could be an approach for analyzing the human-machine interaction in terms of, 
e.g., wearing comfort, support, and motion synchronicity, as well as optimizing 
this interaction in real time based on the analyzed sensory data. For this, force 
sensors need to be embedded in interfaces, which implies the consideration of 
inexpensive sensors to minimize the total purchase price. However, measuring 
contact forces on the wearer is challenging and inexpensive flexible force sensing 
resistors have limited accuracy, repeatability, and stability. This paper evaluates 
the suitability of an interface principle working with two water capsules and two 
embedded piezo-resistive pressure sensors in different test scenarios derived 
from real exoskeletal application examples. Finally, a comparison of the cap- 
sules’ inner pressures reliably detects different load conditions on the interface 
such as centered, edged, and shear forces. Thus, this principle seems to be suita- 
ble for further exoskeletal considerations. 


Keywords: Force Sensors, Low-Budget, Exoskeletons, Human-Machine Inter- 
action, Evaluation, Adaptability. 


1 Introduction 


Exoskeletons have become increasingly popular within the industrial sector with vari- 
ous applications [1]-[3] to support the user by decreasing general workloads or work 
in forced positions, as well as increasing working productivity due to prolonging mus- 
cle fatigue [4]. In principle, the successful application of any support system requires a 
merger of human, machine, and activity [5]. Thus, exoskeletons feature different prop- 
erties based on their morphological structure such as path of force, structural rigidity, 
and actuation [3]. To assess the functionality, suitability, and impact of an exoskeleton 
in a certain context, an evaluation of the system should be conducted first, considering 
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diverse criteria like level of support, kinematics, comfort, and safety [6]. For this, dif- 
ferent methods ranging from biomechanical assessments to questionnaires are usually 
used to get a comprehensive impression of the exoskeleton’s performance [7]. 

The implementation of force sensors in interfaces of exoskeletons may help to con- 
duct a multicriterial evaluation of exoskeletons using only one tool. Statements con- 
cerning level of support, control and dynamics, situation and intention detection, path 
of force as well as the design, fit and comfort of its interfaces are therefore possible. 

If such sensors are incorporated in the interface, the sensory data may be analyzed 
in real-time and initiate corresponding adjustments of the exoskeleton. This may im- 
prove human-machine interaction and acceptance of exoskeletons equipped with force 
sensors. Additionally, this approach includes the use of inexpensive sensors in order to 
keep total cost of the exoskeleton at a competitive level. Overall, it depicts one way for 
exoskeletons to become smarter devices by improving their automatic adaptability to 
different working environments, tasks, and users. 

In principle, the accurate measurement of contact forces on human interfaces is chal- 
lenging [8] and requires diverse sensor characteristics like linearity, low creep, low 
temperature sensitivity, flexibility, and repeatability [9]. When focusing on inexpensive 
sensors for this application, only piezo-resistive force sensors seem to be suitable [10]. 
Diverse approaches have already been made using different Force Sensing Resistors 
(FSRs) [8], [11], however these sensors usually suffer from low linearity, stability, re- 
peatability, high drifts and hysteresis [10], [12]-[14]. In this context, commercially 
available pressure sensor pads are also unsuitable due to their high purchase costs and 
their complexity for further processing sensor data in real-time. 

The use of high-resolution pressure sensors instead of FSR seems to be beneficial 
for measuring surface pressure on interfaces [15], whereby the average pressure on the 
interface divided by the interface region implies the total support force of the exoskel- 
eton [14]. In this paper, different test scenarios derived from real exoskeleton applica- 
tions evaluate the general suitability of a developed exoskeleton interface principle with 
two water capsules and two embedded pressure sensors. These scenarios mentioned in 
chap. 3.2 may simulate possible load conditions on exoskeleton interfaces to assess the 
general human-machine-interaction and reveal recommendations for further actions. 


2 Interface Prototype with Embedded Pressure Sensors 


The developed interface prototype measures pressure conditions within a flexible cap- 
sule with the help of embedded pressure sensors. If a load occurs on the interface, the 
flexible capsule is compressed and its inner pressure rises, which is detected by the 
pressure sensor. Overall, the function of the flexible capsule can be broken into two- 
part: a) pressure detection and b) replacement of foam for bolstering the interface by 
working as a cushion itself. 

In its basic structure, the prototype is inspired by the work of Lee et al. [15], who 
designed a prototype for a pressure sensor system based on a “micro-sensor inside a 
flexible capsule filled with a pressure transmissible fluid”. Their prototype was built 


71 


out of a plastisol bladder as a flexible capsule, laminated on an aluminium plate, using 
hair gel as transmissible fluid, and the pressure sensor MPX2300D. 

Here, the interface prototype uses the above basic construction principle, but a cast 
silicone capsule, water as transmission fluid, the modern high-resolution pressure sen- 
sor MS5803-01BA, and a second aluminium plate for detachable clamping of the flex- 
ible capsule onto the aluminium baseplate. In addition, the prototype is expandable to 
additional flexible capsules with multiple pressure sensors. This may be useful, as apart 
from magnitude and duration, the pressure distribution is of special interest for human 
interfaces, which is caused by changes in human soft tissue or mismatches of exoskel- 
etal and human axes [11], [14], [16]. In addition, the force magnitude may be solely 
assessable via power curves of the exoskeleton actuation or simulated human models. 
In this approach, the prototype does not feature a third chamber in the middle of the 
interface, due to load usually shifting towards its edges [11]. Furthermore, this approach 
is supported by an economic perspective limiting the total amount of sensors. 


2.1 Construction 


The construction of the interface prototype is illustrated in the figure below. The left 
side shows its lateral cross section at sensor level, the right side its total view presented 
in an exploded view drawing. 
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Fig. 1. Construction of the interface prototype with embedded pressure sensors. 


The constructive base of the interface prototype is a milled rectangular aluminium plate 
with the dimensions of 100 x 60 x 10 mm, that represent the exoskeletal interface. For 
later load experiments, holes in the center and the right edge are drilled into the plate, 
and a hitch on the right lateral face is installed. On the interface, the two pressure sen- 
sors are placed in two transfixions on a pressed sleeve and clamped with a crossbar (not 
illustrated in Fig. 1). This connection is sealed with an O-ring on the outer diameter of 
the metal cap of each sensor [17]. 

The used pressure sensor MS5803-01BA, which is based on leading MEMS tech- 
nology and placed on a breakout board for an easier junction, is a piezo resistive, high- 
resolution, low power, and economical altimeter sensor with adjustable resolution. For 
the used resolution of 0.012 mbar, the response time is 8.22 milliseconds. The accuracy 
in the used range is + 1.5 mbar. Due to a gel protection, the sensor can measure pres- 
sures in different mediums like liquids or air. For further technical details, see [17]. In 
this set-up, the sensor runs on an Arduino UNO via °C communication. 
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The flexible capsule needs to be shapeable, resilient, sealed, and skin-friendly, which 
is achieved by the multi-purpose silicone “Dragon Skin 20” [18]. For molding the cap- 
sule, this two-component silicone is mixed, stirred, evacuated to release air bubbles, 
and cast in a milled mold. The benefits of the molded capsule are a return to its original 
form without load, no recorded irritabilities in case of longer skin contact, and an easy 
adaptability of its shape to different dimensions of exoskeletal interfaces. The dimen- 
sions of each capsule are 60 x 47 x 10 mm and the wall thickness is 1.0 mm on the sides 
and 1.5 mm on the bottom and the top. After curing, each capsule is positioned under 
the corresponding transfixion of the interface and clamped by a second milled alumin- 
ium plate with four mounting screws, respectively. After that, the capsule is filled with 
water, vented, and sealed with the pressure sensor by fixing the screws of the crossbar. 


2.2 Calibration of the Pressure Sensors 


First, the prototype is placed on a hard surface, e.g., a wooden table. The calibration 
weights are centered on a 3D printed sheet that is stuck into the center hole on the 
aluminium plate, in order to consistently distribute the load on each silicon capsule. 
The following calibration process is derived from [8]. Under barometric pressure and 
room temperature with relative humidity, the interface is loaded and unloaded in the 
center with three weights (200 g, 350 g, 500 g) three times with a duration of five sec- 
onds each and a sampling rate of 50 milliseconds. Three pressure values after four- 
second measurements are taken and averaged. Based on the respective pressure gradi- 
ent of each load against the pressure without load, a linear three-point calibration is 
made. The quality of the linear regression using this method has proven very sufficient 
(R? sensor! = 0,9997 and R*sensor2 = 0,9999) and even a little improved compared to [15]. 
This suggests that the linear performance of each sensor is very good in this measure- 
ment range. For later interpretation, the load with weights in gram (g) is translated to 
an acting vertical force in Newton (N). 


3 Experiments 


Since Lee et al. [15] already demonstrated the benefits of this measurement principle 
against other commercially available, low-cost force sensors in terms of linearity, tem- 
perature effect, hysteresis, reliability and creep, the key aims of this work are to place 
these within the context of exoskeletons: 

e Is the developed prototype suitable for assessing loads, load distributions, and shear 
forces on an exoskeletal interface? 

e How many pressure capsules are necessary and in what arrangement, to detect dif- 
ferent load conditions reliably? From an economic perspective, the aim is to keep 
the number of sensors to a minimum. 

For this, different experiments are derived from possible application scenarios for an 

exoskeleton for the upper extremities. Each experimental set-up uses varying weights 

and weight positions to represent different support forces of an exoskeleton, or load 
conditions caused by different movements and tasks. 
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3.1 General Test Set-Up 


In order to focus on the principal applicability, the prototype is tested in a controlled 
laboratory setting. In reality, the supportive force of an exoskeleton affects the human 
soft tissue through the interfaces, where the interface distributes the force as surface 
pressure to the contact area on the body. Various load conditions, distributions, and 
shear forces may occur (see Fig. 2). In the lab experiments, this is simulated with dif- 
ferent weights placed on a 3D printed plate and positioned in the center and on the edge 
of the interface. A spring scale simulates shear forces. The human soft tissue is modeled 
with two silicone layers with different shore hardness [19] (upper layer: “Ecoflex 00- 
10”, lower layer: “Dragon Skin 20”). 
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Fig.2: Derivation of the general test set-up and scenarios from real exoskeletal 
applications on the example of the upper arm. 


3.2 Test Scenarios 


Test scenario A “Equal Continuous Constant Load” describes a task that is per- 
formed for a prolonged period of time in the same physically demanding posture (i.e. 
the complex electronic cable insertion in car doors, which is performed overhead/above 
shoulder level). This is tested with a constant load of 4.9 N placed only in the center of 
the interface for 120 seconds. Lee et al. [15] tested such a scenario for 144 seconds. 
Test scenario B “Continuous Decreasing Load” considers a continuous load on the 
center resp. edge for a prolonged period, which gradually decreases over time. One 
example may be the emptying of goods from a large box/container and placing them 
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separately onto shelves or into different storage spaces, so that the weight of the load 
carried decreases. Another example is the installation of pipes on a ceiling with clamps, 
so that the weight of the pipe decreases with every fixed clamp. In the tests, the interface 
is loaded with 5.9 N to begin with, and then slowly decreases to 5.8 N, 5.4 N, 4.9 N, 
and 3.9 N after six seconds each. The respective gradient deliberately varies. 

Test scenario C “Repetitive Constant Load” represents a repetitive task at head 
level or above, e.g., driving screws in a ceiling, with the same weights of tools or other 
components. If users raises their arms, the exoskeleton may support them, hereby ap- 
plying surface pressure to the human tissue. If users lowers their arms, no pressure is 
applied due to the lack of load on the upper extremities. In the simulation, the alumin- 
ium plate is repetitively loaded five times on the center resp. edge for three seconds 
with 3.9 N. A pause of three seconds is conducted in between. 

In the test scenario D “Repetitive Varying Load”, the ability of the prototype of 
measuring varying repetitive loads is evaluated. In practice, these loading conditions 
may appear, if the user of an exoskeleton needs to randomly handle light and/or heavy 
packages (i.e. working in logistics). The support of especially active exoskeletons is 
usually adaptable, which may lead to varying loads. In the test set-up, the interface is 
consecutively loaded on the center resp. edge with 4.9 N, 0.2 N, 3.9 N, 1.5 N, and 4.9 N 
for a duration of three seconds. A pause of three seconds is conducted in between. 

Test scenario E “Repetitive Varying Shear Forces” evaluates the ability of the 
prototype to assess shear forces during motion, that may be a cause for friction and may 
even lead to skin abrasions. Shear forces may be an indicator for a poor fit and conse- 
quently lead to rejection of the exoskeleton. In the test, the interface is vertically con- 
tinuously loaded with 4.9 N and in addition repetitively loaded horizontally in longitu- 
dinal resp. lateral direction of the interface, with a varying traction force of 2.0 N, 1.0 N, 
2.9 N, 4.9 N, and 2.9 N. A shaped metal crossbar prevents any rolling of the interface 
to focus only on the influence of horizontal motions of the interface on each capsule 
pressure. 


4 Results 


For every test scenario, the different sensory data are stored and plotted in Matlab 
R2018b and presented in Fig. 3. Based on the inner pressure and the calibration of each 
sensor, the respective load conditions may be concluded. 

Both sensors accurately measure near the same pressure, if the interface is equally 
loaded with the same weight for a prolonged period of time (see scenario A). Both 
sensors also measure similar pressure for equal repetitive loads (see test-retest scenarios 
C and D). In case of an equal load, both sensory values add up to near the applied load. 
If an unequal load occurs, the inner pressure of the capsule underneath the load 
measures the applied load and additionally rises with almost the same gradient like the 
equivalent of decrease in pressure occurring to the other capsule (see scenario B, C, D). 
Test scenario B shows that the sensors may even detect marginal changes in load as 
small as 0.1 N, and further, measure varying consecutive loads with high accuracy (see 
scenario D). 
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Fig. 3: Results of the different test scenarios. The abscissa shows the passage of time 
and the ordinate the inner pressure in the capsules measured by the pressure sensors. 
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However, small loads may be detected as a rise in the inner pressure but may also be 
wrongly interpreted as a negative or decreasing force (see scenario D). If horizontal 
shear forces longitudinally act on the interface, the value of the anterior sensor to the 
location of applied load rises slightly, while the value of the posterior sensor decreases 
at a larger rate — this may be true even for small forces. This effect does not appear for 
laterally acting shear forces (see scenario E) and all vertical loads (see scenario B-D) 
on the interface. 


5 Discussion 


An analysis of the results shows that the interface prototype has a good stability, re- 
peatability, and accuracy in its measurements. Even marginal load changes may be de- 
tected, however, their interpretation may be prone to error, due to the constant of the 
linear regression, on which the calibration of the sensors is based. Thus, although a 
small load is usually detected, it may be wrongly interpreted as a small decrease in load. 
However, a comparison of both sensory data may help to identify and objectively in- 
terpret the following load conditions on the interface, which may include recommen- 
dations for further actions. 

For the following ratios, the respective interpreted load identification is presented. 
The “X” represents a by an unknown factor changing variable, dependent on the corre- 
sponding load condition: 

e pressure 1 ~ (applied load)/2 and pressure 2 = (applied load) /2: 
The load is near equally distributed. No need of action, the interface fits well. 
e pressure 1 ~ applied load + X and pressure 2 = —X: 
The load is unequally distributed, leading to load peaks on the surface area covered 
by the capsule with a positive value (here pressure sensor 1). The exoskeleton does 
not fit well and there may be a possibility for perceived discomfort. Adjustments 
are recommended. 
e pressure 1 ~ — applied load + X and pressure 2 = X: 
Horizontal shear forces act longitudinally upon the interface, which may lead to a 
displacement of the interface. The exoskeleton does not fit well and the possibility 
of abrasions exist. Adjustments are recommended. 
With two capsules, the prototype is only capable of detecting load distributions and 
shear forces on the longitudinal axis of the interface. For statements on the lateral axis, 
two additional flexible capsules must be implemented on the interface in this direction. 
This would enable the detection of any spatial load on the interface. At this point, the 
benefit of an additional centered capsule requires further investigation. 

However, the interface prototype depends on the barometric pressure, as the gradient 
against the barometric pressure is translated into respective load. Thus, the prevailing 
barometric pressure needs to be detected prior to each measurement to ensure accurate 
results. In addition, the gravity of water in the capsule has an influence on the inter- 
preted load as well. This may occur, if the interface spatially rotates while performing 
certain tasks and the sensor does not remain in its vertical orientation, as it was in the 
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calibration. Here, the implementation of an Initial-Measurement-Unit (IMU) on the in- 
terface may help calculate an offset. Alternatively, air may be used to replace the water 
medium, however this could lead to other limitations and challenges, particularly in 
sealing the entire interface airtight. 

Finally, it must be acknowledged that the test set-up itself presents some limitations. 
First, the model of the human tissue used in the above study is fairly consistent, which 
is not always the case when working with human subjects due to factors such as varying 
tissue and bone structures. The loads used may also not be representative for all possible 
exoskeletal scenarios. However, as long as a mechanical load test of the interface pro- 
totype is pending, the loads are limited to 5.9 N in total, for safety reasons. The planar 
shape of the interface prototype and the tissue model are also abstract, acknowledging 
curvatures and uneven surface structures in reality for both. Finally, the environmental 
influences on interfaces vary in terms of temperature and humidity, which are not con- 
sidered in the controlled laboratory setting. 


6 Conclusion 


In this paper, the suitability of an interface prototype with embedded pressure sensors 
for assessing and optimizing the human machine interaction in the context of exoskel- 
etons is evaluated. In five test scenarios, it has been shown that this prototype is capable 
of reliably detecting different load conditions, between an interface and a model of hu- 
man tissue. 

In the future, two main areas should be focused on: the construction and the analyt- 
ical ability of the interface prototype. First, a load and repeatability test of the entire 
prototype should be conducted, especially of the silicon capsule. Efforts should also be 
made to install and test this principle on curvatures, as most exoskeletal interfaces are 
not planar. On the other hand, the prototype should be tested on humans under real 
conditions and embedded in exoskeletal interfaces, whereby the influence of wearers’ 
body temperature and sweat as well as the ability of replacing foam bolstering with the 
flexible capsule may be a matter of interest. The favored level of support for diverse 
tasks or critical pressure levels in terms of subjective perceived comfort may also be 
considered. At this point, the evaluation may be extended to health issues by looking at 
blood flow [20] or oxygen supply beneath the pressured surface [21] in association with 
near infrared spectroscopy (NIRS). 
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Abstract. Due to their many advantages, flexible structures are increasingly be- 
ing used as guide and transmission elements in handling systems. Prismatic solid- 
state joints with a concentrated cross-sectional reduction are predominantly used 
as flexure pivots for both microscopic and macroscopic designs. A transfer of 
these geometries to applications in cryogenic working environments is not easily 
possible at temperatures below -130 °C due to the changed material properties. 
In this paper, the further development of swivel joints as cascaded solid state 
joints for such a cryogenic environment is illustrated by the targeted adaptation 
of certain joint parameters and dimensions. By means of a comprehensive FEM 
simulation, it can be shown how the influence of specific parameters affects 
movement accuracy, process forces and shape stability and to what extent these 
geometric parameters influence each other in their effect. 


Keywords: compliant mechanisms, flexure hinges, cryogenic workspace. 


1 Introduction 


In today's industry, automation is an omnipresent factor or will be, even in niche appli- 
cations such as cryopreservation. Nevertheless, Manual handling of biological or toxic 
samples is still the rule in research facilities. Such samples are conserved in so-called 
biobanks. In their simplest form, these are Dewar vessels are cooled with liquid nitro- 
gen. However, large semi-automated systems are also used, mainly for an indefinite 
storage time. Even in such systems, the samples are still stored, rearranged or trans- 
ferred by hand using bulky protective clothing and equipment. Doing so involves a 
significant risk of injury due to cold burns for the users on the one hand. On the other 
hand, the sample integrity can be compromised by any kind of heat exposure during the 
handling process. To overcome the dangers of manually handling test tubes while cir- 
cumventing the technical challenges a cryogenic work environment presents, a full au- 
tomation is desirable. The logical consequence is to develop a robot based handling 
system. 

As part of the DFG funded project “Methods for the automation of handling pro- 
cesses under cryogenic environmental conditions”, a parallel robot is being developed 
at the Institute of Assembly Technology at Leibniz Universität Hannover to implement 
such full automation. Especially the passive joints of the robot have to meet very high 
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demands: The extreme temperatures of below -130 °C do not allow the use of classic 
rigid body systems such as ball joints due to the freezing of lubricants or the clamping 
of components due to cold shrinkage. To avoid these disadvantages, solid-state joints 
in the form of cohesive swivel joints are used. Due to their monolithic construction, 
clamping is not an issue since no parts moving against each other are involved. In ad- 
dition, the use of lubricants is also unnecessary. 

Due to these and other advantages, joints with concentrated resilience designed as 
monolithic coupling mechanisms have already become widely used in technical and 
industrial handling technology. In these one-piece constructions, an inherent compli- 
ance is achieved exclusively by reversible deformation of certain areas. Thus, these 
mechanisms can be used for small angular deflections or displacements instead of con- 
ventional form- or force-paired pivot hinges. The most common application to date has 
been prismatic solid state joints with areas of reduced cross sections as bending points. 
These mechanisms are mainly found in precision systems in precision mechanics and 
metrology as well as in applications in precision assembly, i.e. areas in which high 
demands are placed on positioning accuracy and resolution [1—4]. Applications in mac- 
roscopic constructions and for big angular deflections are still the exception. 

Solid-state joints are material-coherent mechanisms and their ability to move is 
based on the deformation of the entire joint or on the deformation of areas with a con- 
centrated reduction in cross-section. Therefore, their bending range is severely limited 
to a few degrees by the maximum allowable stresses in the component. In addition, a 
rotational movement of a material-locking joint always causes a load-dependent dis- 
placement of its axis of rotation, which has a negative effect on the positioning accuracy 
of the mechanism. It is therefore always desirable to achieve the smallest possible ratio 
of maximum stress to the rotation achieved, while at the same time minimizing the 
displacement of the axis of rotation. To achieve this, increasingly complex geometries 
have been developed in the recent past, or an increasing number of simple solid-state 
joints have been installed in a common structure. 


1.1 Related work 


In contrast to cohesive joints with only one area of concentrated flexibility or with 
multi-unit flexible coupling mechanisms, the structural and dimensional synthesis of 
complex flexure pivots has not yet been discussed in detail. The literature mainly de- 
scribes constructions with concentrated cross-section reductions in rectangular or cir- 
cular contours, as well as the effect of these contours on the achievable range of motion 
[5-17]. There is also work on the geometric design of complex swivel joints on the 
basis of interlinked tape spring joints [18], which allow large angular deflections. The 
relevant influences on the geometric design of such joints with regard to their ability to 
move along the main axes and parasitic axes have hardly been investigated. The project 
”Methods for the automation of handling processes under cryogenic environmental 
conditions”, however, was preceded by the work of Raatz et al. [19-21]., in which the 
use of cohesive joints in parallel robot structures and the kinetic and kinematic behavior 
of these joints were examined in detail. The use of solid-state joints in a cryogenic 
working environment has not yet been investigated. The work on the use of flexible 
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joints for use in aerospace engineering has been described [3, 4, 22], but not for the 
environmental conditions envisaged in this project. 

Following on from the work of Robert Fowler [4] and Simon Heinein [22], in this 
paper the finite element method (FEM) and the ANSYS program will be used to present 
a joint geometry that is capable of achieving a required angle of rotation in a cryogenic 
working environment. The maximum permissible stresses within the components are 
never allowed to exceed a certain value, so that the deformation of the joint is com- 
pletely reversible at all times. In addition, by varying certain design parameters, their 
influence on the necessary process forces, the undesired deformation along certain axes 
(henceforth called parasitic axes) and the effect of these parameters on each other will 
be investigated. 


1.2 Materials and methods 


The aim of this work is the further development of cascaded solid state joints based on 
the work of Robert Fowler [4], who in turn bases his work on the Butterfly Pivots by 
Heinein et al [22, 23]. Especially the Flex-16 large-displacement compliant hinge offers 
a good starting point for the development and investigation of own joint geometries 
(Fig.1). Due to the continuum mechanical problem, a sequential approach of concept 
synthesis and constructive development of the design is not easily transferable to the 
existing compliant hinges. The displacement of the axis of rotation allows only limited 
formation of a replacement model based on rigid state mechanisms. In addition to the 
desired rotational motion, a force-deformation characteristic along the main and para- 
sitic axes as well as the shape stability must also be taken into account in the structural 
development of compliant mechanisms. The procedure here follows the idea of a to- 
pology-optimized structure. The relevant load paths and most stressed points of a pro- 
totype are identified. Then the material distribution is varied within the framework of 
certain constructive and production-related restrictions and the change of the load cases 
is evaluated afterwards. In particular, it will be investigated whether the motion and 
deformation behavior can be represented as a function of the geometry parameters and 
whether there are interdependencies between the geometry parameters with respect to 
their influence on the motion and deformation behavior. For this purpose, four criteria 
are to be considered qualitatively and quantitatively as result variables: 


e The maximum von Mises stresses occurring in the critical areas when a defined 
bending angle is reached 

e The deviation of the rotary axis from the initial position 

e The deformation of the entire joint along two (parasitic) axes perpendicular to the 
axis of rotation. 

e The forces required to reach the defined bending angle 
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Fig. 1. Prototype Geometry based on Robert Fowlers Flex-16 large-displacement compliant 
hinge 


The simulation of the joints in ANSYS is to be carried out under the conditions typical 
for a cryogenic working environment, taking into account the changed material prop- 
erties of the joint material at temperatures below -130 °C. However, since these tem- 
peratures are reached in a biobank by cooling with liquid nitrogen, the components 
undergo the danger of being exposed to much lower temperatures. For the simulations, 
the adjusted material parameters Young’s Modulus and transverse contraction number 
are used from tabulated sources [24, 25] for a temperature of -196 °C, which corre- 
sponds to the boiling temperature of liquid nitrogen and includes the case that the joints 
are immersed in the liquid. 


2 Prototype geometry 


The starting point of the joint construction is the Flex-16 joint, which was presented in 
multiple variations in [4]. After initial estimations and simulations, the overall dimen- 
sion was fixed at 50 x 50 x 10 mm’. This limitation was chosen because of the space 
available in the final construction of the parallel robot, which will be examined in more 
detail in other works. For a better understanding, a design of this robot and a Cardan 
joint consisting of six flexure pivots are shown in Fig. 2. In [4] it has already been 
shown, that this flexible mechanism is able to achieve a bending angle of 90° at room 
temperature, if titanium was used as the base material. 
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Fig. 2. a) Parallel robot structure and b) Cardan joint 


2.1 Simulations of the prototype geometry 


The rotational movement of the joint in the ANSYS simulation is realized by fixing the 
joint at one end and applying various forces at the other end to investigate all possible 
load cases. Care has been taken to ensure that the maximum equivalent stresses do not 
exceed the maximum yield strength that the material can withstand under cryogenic 
conditions. This is made possible by the fact that the bending angle to be achieved for 
the entire joint is divided between the 16 serially linked thin sections of the joint. The 
simulation results of the bending are shown in Fig. 3 a). From this preliminary exami- 
nation, the most heavily loaded or most heavily deformed sections were determined. 
As can be seen in Fig. 3a), the sections 3 and 4 are most strongly involved in the rota- 
tional movement about the intended bending axis. In Fig. 3 b), the sections 1 and 2 
show the strongest local deformation in positional deviation along the parasitic axes 
under a tensile load. 


A: Statisch-mechanische Analyse 
Vergleichsspannung 
Typ: Vergleichsspannung (von Mises} 
Einheit: MPa 

Zeit: 1,0809 1 
07.02.2020 18:20 


352,35 Max 
313,2 
274,05 
2349 
195,75 
156,6 
117,45 
78,304 
39,155 a) 


0,0054805 Min 


Fig. 3. Flex 16 under a) tensile load and b) bending load 
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While the bending movement around the axis of rotation should be achieved with as 
little process force as possible, it is desirable to stiffen the joint in the direction of the 
parasitic axes as much as possible by adjusting the geometry. The geometry parameters 
described in the following and their effects on the result variables mentioned in section 
1.2 have particularly this stiffening as their goal. After several simulations with differ- 
ent materials, the choice of material falls on titanium. The minimum 20° bending angle 
of the joints required for robot design could not be achieved with any other material 
within the limited installation space. 


3 Development of the joint geometry 


The flexure joint developed here will later be manufactured from the titanium alloy 
TiAI6V4 in a laser sintering process. This choice was made, because it is by far the 
most common titanium alloy available for industrial additive manufacturing processes. 
The hinge consists of 16 individual thin sections arranged around a central connection 
point. Four areas with a comparatively large cross-section - so-called shuttles - are ar- 
ranged around the fulcrum to ensure axial stability. The thin sections form six "wings". 

The first geometric parameter to be varied is the thickness of the material. It is ex- 
pected that an enlargement of the cross-section h (see Fig. 4b) will lead to a stiffening 
of the joint. The extent of this stiffening is to be investigated for a range between h=0.8 
mm and h=1.2 mm in 0.1 mm steps. The second parameter describes the shortening of 
the thin spots called wings. Here it is assumed that a shortening of the wings will lead 
to a stiffening of the joint. The wing proportions are shown in here Fig. 4c). Their length 
lis varied between /=15 mm and /=5 mm in steps of 2 mm. The stiffening by an en- 
largement of the cross section and the shortening of the thin sections mainly involved 
in the bending are logical steps. The last geometry parameter to be varied is derived 
from the results of the prototype geometry from Fig. 3b. As can be seen, the upward 
bending of the wings seems to be the main reason for the deformation in the y-direction. 


To minimize this deformation, small corner shuttles are provided in 
— = u. . nn me e ——— | FE 
D 


a) 


c) 


Fig. 4. a) Prototype proportions, b) widened cross-section h, c) shortened wing length /, d) in- 
creased radius for corner shuttles r 
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the wings, whose radius r is varied between 7=0.4 mm and 7=3.2 mm in steps of 0.2 
mm (see Fig. 4d). The minimum variation of the geometries is based on the manufac- 
turing parameters of a laser sintering process, which can generally be guaranteed by the 
manufacturers [26, 27]. The number of possible parameter combinations resulting from 
this is two hundred and twenty-two simulations. 

Depending on the selected geometric proportions, different bending angles can be 
achieved before the deformation causes critical stresses in the components. Since a 
minimum angle of twenty degrees for each solid state joint is sufficient for the parallel 
robot structure to be realized, it was checked whether all joint variations can represent 
this angle, and if so, which maximum angle can be achieved by all joints. The prelimi- 
nary investigations showed that the smallest bending angle that can be achieved by 
elastic deformation of all joints is 27.75 degrees. This angle was assumed as a fixed 
value for the simulations. Thus, the necessary applied bending force as well as the max- 
imum von Mises stresses occurring can be used as target values. In addition to the bend- 
ing, the joints are also loaded with 16 N for tension in the y-direction and with 16 N for 
shear in the x-direction in order to investigate a deformation along the parasitic axes. 
The target values for these load cases are the deviations from the initial position along 
these axes. 


3.1 Simulation of the joint geometry 


The basic geometry of the flexure pivot based on the Fowler model [4] is created with 
solid works. The geometry parameters to be adjusted are then inserted and the model 
parameterized in ANSYS. The mesh quality of the model significantly influences the 
calculated stiffness in the simulation [28]. In order to find a compromise between the 
necessary simulation time and sufficient accuracy, the number of nodes used per model 
is limited to 8200. This number resulted from the specification that every element has 
an edge length of no more than two times the material thickness. The aim was a multi- 
zone mesh. Areas of the solid state joint that have a simple geometry (mainly cuboids 
or circular sections) can be swept and thus meshed very evenly. For the areas where 
these joint sections meet, a finer mesh structure has been strived for. Here a hex-domi- 
nant mesh was preferred, if possible. The use of linear tetrahedrals was avoided as this 
element class usually implies too high a stiffness. The skewness, which should not ex- 
ceed the value 0.5, was considered as a measure for the quality of the mesh. These 
restrictions were met for all simulated joints. Due to the quality of the net, it was pos- 
sible to forgo center nodes, which can also cause excessive stiffening of the simulated 
component [28]. A Sparse Matrix direct solver was used with 40 substeps and a mini- 
mum increment of 0.51107 for the convergence criteria (CC). 
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Graphical evaluation of the results 


PEE 


Fig. 5. Deflection of the flexure pivot und a) bending load, b) shear load, c) tensile ae 


Two hundred and twenty-two simulations were carried out in this way. Based on this, 
a comprehensive examination and analysis of the data was carried out. Fig.5 shows an 
example from the simulation with the corresponding images of the deformed joint un- 
der the given load cases. The consideration of the target values (see section 1.2) shows 
a trend in the effect of the varied geometry parameters. Fig. 6 shows the maximum 
stresses as an example. 


Max. Stresses at 27,75° 
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Fig. 6. Maximum stresses, sorted by the varied geometric Parameters 


The course supports the assumption that the geometric parameters are not independent 
of each other with respect to the effect they have on the target variables. It can also be 
seen that the effect of the changed parameters material thickness, wing length and cor- 
ner shuttle have a different influence on these target values. The widening of the cross- 
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section has by far the strongest influence on our target values, whereby the increase of 
the necessary bending forces is rated as negative, and the reduction of the deflection of 
the joints along the parasitic axes is rated as positive. Analogous conclusions result for 
the wing length and the radius of the corner shuttles in descending significance. Here, 
too, it can be seen that the effect of the parameter change is not linear, which again 
supports the assumption that the geometric parameters are not independent of each 
other. In order to investigate this in more detail, the results of the simulations were 
divided into different classes. The simulated feature carriers are first divided into six 
classes sorted according to the cross-section thickness and then the target values are 
displayed as a function of the wing shortening. The corner shuttles are set to a radius of 
0,4 mm. As can be seen in Fig 7, the graphs do not run parallel to each other. Analogous 
to this procedure, six subclasses are formed for the wing length and then the target 
values are displayed as a function of the radius of the corner shuttles. This time the 
material thickness is set to 1 mm. The result forms a similar picture as in the previous 
consideration. The course of the curves is not constant between the subclasses. 


Maximum stresses. Class: Cross-Section Widening; Constant: Corner Shuttle 
Radius 


1100 


Max. stress [MPa] 


400 | 


2 3 4 
Wing shortening 
-~ X-sect. widening 0 [mm] --X-sect. widening 0,1 [mm] -+-X-sect. widening 0,2 [mm] 


— X-sect. widening 0,3 [mm] —- X-sect. widening 0,4 [mm] 


Fig. 7. Class comparison: cross-section widening 


5 Analysis of the results 


The graphical evaluation of the results suggests that there is no linear correlation be- 
tween the target values and the individual geometry parameters. For a multivariate anal- 
ysis approach, a regression analysis based on a polynomial is suitable. Based on Fig.7 
however, it is assumed that the parameters themselves depend on each other in their 
effect on the target parameters and therefore no linear regression model can be used as 
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a basis. At this point, however, qualitative statements about the influence of the geom- 
etry parameters on the target variables can already be made. These are summarized 
below. A further procedure on the basis of the obtained data is discussed in the outlook. 
Based on the results obtained so far, the following qualitative correlations can be con- 
cluded: 


e The three selected geometry parameters material thickness, wing length and corner 
shuttle radius all have a negative influence on the target parameter "bending force" 
and a positive influence on the target parameter "deviation along the parasitic axes". 

e The significance of the influence varies between the geometry parameters 

e The geometry parameters are not independent of each other. They influence each 
other in their effect on the target parameters. 

e It appears that the decrease in wing length and the increase of the corner shuttles 
both have a positive influence on the maximum stresses. The reasons for this are not 
yet clear, but an assumption can be made, that this is the result of a better overall 
distribution of the maximum stresses throughout the flexure hinge. 


Because of the data obtained, the following quantitative statements can be made, even 
if the formula-based relationship is still unknown. Fig.8 sums the results up graphically: 


e A cross-section widening of 0.4 mm increases the necessary bending forces for 
reaching the defined angle by 711 % from 17.7 N to 127,26 N. The position deviation 
in the x-direction due to the shear load is reduced by 85% from 1.39 mm to 0.20 mm. 
Similarly, the position deviation in the y-direction due to the tensile load is reduced 
by 87% from 0.89 to 0.12 mm. The maximum stresses are increased by 121%, from 
482 N/mm to 1065 N/mm. 

e Shortening the wings by 10 mm increases the bending forces required to achieve the 
defined angle by 42 % from 17.7 N to 25,08 N. The position deviation in the x- 
direction due to the shear load is reduced by 45% from 1.39 mm to 0.76 mm. Simi- 
larly, the position deviation in the y-direction due to the tensile load is reduced by 
46% from 0.89 mm to 0.48 mm. The maximum stresses are decreased by 1%, from 
482 N/mm to 478 N/mm. 

e Increasing the radius of the corner shuttle by 2.8 mm increases the bending forces 
required to achieve the defined angle by 20 % from 17.17 N to 21.25 N. The position 
deviation in the x-direction due to the shear load is reduced by 30% from 1.39 mm 
to 0.96 mm. Similarly, the position deviation in the y-direction due to the tensile load 
is reduced by 42% from 0.89 mm to 0.52 mm. The maximum stresses are decreased 
by 14%, from 482 N/mm to 416 N/mm. 

e In the extreme case that all geometry parameters are set to the maximum of their 
values, the necessary bending forces for reaching the defined angle increases by 
1052 % from 17.7 N to 204 N The position deviation in the x-direction due to the 
shear load is reduced by 94% from 1.39 to 0,09 mm. Similarly, the position deviation 
in the y-direction due to the tensile load is reduced by 95% from 0.89 to 0.05 mm. 
The maximum stresses are increased by 91%, from 482 N/mm to 919 N/mm. 
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Fig. 8. Summary of the results 


6 Conclusion and outlook 


Due to the assumed non-linearity and lack of independence of the varied geometry pa- 
rameters from each other, no formula-based procedure for the optimal design of the 
flexure pivot described here can be proposed at present. However, it could be shown 
that the geometric adaptation of the joint allows a significant reduction of deviations 
from the nominal position without limiting the functionality or exceeding the critical 
comparative stresses. In the following, further analyses of the existing database will be 
carried out. This includes non-linear regression analyses as well as optimization strate- 
gies. In addition, a characteristic value is to be developed which describes the useful- 
ness of the geometric adaptations as a ratio of the positive and negative effects. If this 
procedure proves to be promising, further joint geometries will be created and evalu- 
ated. 
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Abstract. The current market offers a wide range of different robots and end 
effectors from a variety of manufacturers and in different versions, where the 
mechanical compatibility between the components has to be ensured. A large part 
ofthe flange patterns ofthe end effectors offered are not designed to conform to 
standards, which is one ofthe reasons why adapter plates must be used to ensure 
compatibility. Particularly in the early phases of engineering, the weight of the 
adapter plate and the extension of accessibility can be taken into account in the 
robot configuration when generating an adapter plate. In order to enable the gen- 
eration of an adapter plate as early as possible, automated generation offers a 
great advantage and can contribute to the reduction of engineering effort and thus 
to time savings and cost reduction. This paper describes a general design rule for 
the generation of adapter plates and, on this basis, implements the generation of 
CAD geometries of the adapter plates using Constructive Solid Geometry. The 
results of this paper are the starting point for the implementation ofa web service, 
which will be integrated into the ROBOTOP project. 


Keywords: Flange patterns, construction instruction, Constructive Solid Geom- 
etry. 


1 Introduction 


Industrial robots and end effectors are offered by a large number of manufacturers and 
in an abundance of design variants. For example, 422,271 robots were delivered world- 
wide in 2018, which represents an increase of 15 percent over the previous year. Similar 
figures are predicted for 2019. [1]. The market situation for end effectors is similar, with 
the four best-selling end effectors manufacturers alone (Schunk, Schmalz, Destaco and 
Festo) currently offering over 5,400 different grippers [2-5]. In order to ensure compat- 
ibility between the individual components across manufacturers, the interfaces between 
robot arms and end effectors must be clearly defined. This should make it easier for the 
customer to configure robot systems and ensure a lower knowledge barrier for the com- 
bination of different robot solutions. Small and medium-sized enterprises (SMEs) in 
particular can be supported by this in production automation. Because of its relatively 
low degree of automation, the SME market offers a particularly high potential for future 
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automation through robot systems. Decisive factors with the implementation of such 
robot systems are the high engineering expenditure and the high costs. Thus, many ro- 
bot solutions cannot be economically represented for system integrators up-to-date or 
only to a very limited extent. 

The ROBOTOP project addresses this problem by developing an online configurator 
for robot systems. Within the scope of this research project, the mechanical interfaces 
of the end effectors and robot arms are considered. If the components to be assembled 
have different flange designs, they have to be brought together with an adapter plate. 
In addition to bringing the flange patterns together, the adapter plate also offers the 
possibility of changing the offset position of the end effector on the robot system. Thus, 
on the one hand, the end effector can be realized in the extension of the axis of the 
robot, a lateral offset or a rotation of the end effector is also possible. The generation 
of these adapter plates is currently carried out mainly by the manufacturers when the 
end effectors are delivered or by the customer himself. Here, the development of special 
adapter plate shapes is combined with relatively high effort and is not based on any 
standardized process. Due to the design of the adapter plates from quantity one at the 
customers themselves, the adapter plates are usually oversized for safety reasons. This 
leads to an overall higher dead weight of the adapter plate and thus to a reduction of the 
possible payload of the robot. In addition, the increase in reachability due to the offset 
of the end effector with the help of the adapter plate is often not used at all or only in 
the later phases of the planning process. Thus, the generation of an adapter plate at the 
time of the configuration of the robot system would allow a reduction of the engineering 
effort and an optimization of the robot system selection during the planning of the robot 
system. 

Within the scope of the ROBOTOP research project, the generation of the adapter plate 
is to be automated. This process has to be standardized. A generally valid procedure for 
the generation of an adapter plate is described in this paper and a type code developed 
for this purpose is listed. 


2 Flange patterns 


In order to ensure the compatibility between the flange designs and to guarantee the 
implementation of the robot systems across manufacturers, the interfaces must be stand- 
ardised and associated standardisation introduced. If one considers the standardisation 
situation in the area of robot flanges, ISO ISO 9409 - 1, ISO 9409 - 2 and DIN ISO 
29262 should be mentioned here[6-8]. 

ISO 9409 - 1, published in 2004, forms an important basis for ensuring the inter- 
changeability of manually mounted end effectors in the form of round mechanical in- 
terfaces. The application of this standard is not only limited to industrial robots, but can 
also be applied to hand lifting devices such as insertion devices, manipulators or tele- 
operators. Here decisive characteristic values (among other things type and number of 
thread and fixing holes, diameter specifications) are defined, which allow a holistic 
description of the flange patterns. 

The second part of the DIN ISO standard 9409 defines the main dimensions, the 
designation and the marking of shafts of the mechanical interfaces of industrial robots. 


97 


In contrast to ISO 9409 - 1, the ISO 9409 - 2 is designed for relatively low load capacities 
and is suitable for applications in which end effectors move in small gaps. Furthermore, 
DIN ISO 29262 defines microscopic interfaces for end-effectors and handling equip- 
ment in the field of micro and precision engineering. In addition to the mechanical in- 
terface, the position and classification of electrical and fluidic coupling elements is also 
included. 

A closer look at the robot systems available on the market reveals that the ISO 9409- 
2 and DIN ISO 29262 standards are rarely used. This could primarily be due to the fact 
that the market for robots with such low load capacities serves a niche in which special 
designs are used for the respective application, in which the adaptation to the respective 
robot flange does not represent a large additional expenditure. 

On the other hand, the ISO 9409-1 standard is widely used. Studies of LPS carried 
out that about 81% of all flanges of the seven largest robot manufacturers (ABB, Kuka, 
YAMAHA, Denso, Kawasaki, Mitsubishi and Yasakawa) are designed according to 
this standard. Thus, a relatively high degree of standardization can be observed here. 
The situation is different for the end effectors. A look at the five largest manufacturers 
(SCHUNK, Schmalz, Destaco, Festo and FeRobotic) shows that only 21 % of the end 
effectors comply with the standard. 

It is noticeable that the Cobot gripper flanges in particular exhibit a very high degree 
of standardization. Reasons for this could be, among others, the fundamentally higher 
degree of standardization in the area of human-robot collaboration, the increased safety 
requirements and a higher requirement for weight and design, which does not permit 
an additional adapter plate, which would mean additional weight. 

In conclusion, it can be said that only ISO 9409-1 is used in industry. This standard 
is mainly used in the area of robot arm flanges, but only to a small extent for end effec- 
tors. Furthermore, this standard only defines round flange patterns, where rectangular 
flange patterns are also available in significant numbers on the market. This means that 
there is neither a generally valid description of robot flanges nor does it consistently 
comply with the existing standards. As a result, there is an inconsistent image on robot 
flanges, both on the robot side and on the end effector side. Thus, adapter plates are still 
an important tool in the field of robotics to connect these components with each other. 
At the same time, it becomes clear that for efficient and automated generation, a clear 
design rule must be generated. 


3 Automated adapter plate generation 


Within the scope of ROBOTOP it is necessary to be able to offer a total solution of a 
robot system for the customer, whereby primarily industrial robot solutions were con- 
centrated, which cover small to medium sized payloads. The weight of the adapter plate 
as well as the extension of the accessibility by the adapter plate should already be con- 
sidered in the design of the robot system. Thus, the feasibility of the robot system can 
be better validated at a very early stage, which leads to a reduction in the engineering 
effort and thus to a shortening of the implementation times of the system. 

The basic procedure for generating an adapter plate is shown in Fig. 1. Here it is first 
necessary to select a robot system according to the requirements of the SME. The 
ROBOTOP configurator can be used for this purpose, for example. Then the flange 
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images have to be defined. Here the contact surfaces of the flange and the boron extin- 
guishers for fixing the adapter plate or the end effector have to be defined. The infor- 
mation can be taken either from construction drawings, CAD models or, in the case of 
standardized flange pictures, from the product description. This step decides whether 
or not an adapter plate is required for compatibility of the selected components. In prep- 
aration for generating the adapter plate, the requirements for the adapter plate must be 
defined. Afterwards the automated generation of the adapter plate can be performed. 
An iteration then searches for a new selection of the robot system and repeats the gen- 
eration process. Finally, an optimized selection of possible robot system configurations 
is suggested. In this paper we will concentrate on steps two to four. 


=! @. 
A ‘ie: 
> Flange pattern Requirements A 
Robot syst l ant ss Adapter plat ti 
definition ee 


Fig. 1. Procedure for iterative adapter plate generation. 


There is currently no standardization and no tool that would describe or implement the 
generation of adapter plates. In order to be able to offer an efficient solution for manu- 
facturer-independent and non-standard flange designs, a clear definition of the con- 
struction instruction for adapter plates is necessary. Building on this, the corresponding 
geometry can now be generated using Constructive Solid Geometry. 


3.1 Design specification for the generation of adapter plates 


The unique construction instruction forms the basis for the effective and automated 
generation of an adapter plate. As already mentioned, there is currently no standardiza- 
tion in this area that could be used for this purpose, so it is necessary to define this 
design rule. A generally valid definition was made in the context of the research work 
on the LPS, from which the type code shown in Figure 1 results. In this case, adapter 
plates that connect a robot arm with a gripper were initially used. Furthermore, addi- 
tional kinematics or sensors in the adapter plate were not considered. 
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Each adapter plate consists of two or more flanges which have drilled holes for mount- 
ing and a structure for connecting these flanges. Optional feedthroughs or brackets for 
power and information interfaces can be attached to the worktop. All the relevant pa- 
rameters are briefly listed below. 


F-A-N] [B]-M-Al} {X-M-Z-RX-Rvu-Rz} -0-0 s] {[V)-[F]} 
f f L J L J 
EZ <r 
number of holes connection structure progression 
+ structure structure length of 
regular orientation and position orientation in orientation in the line 
F of the flange patterns in degrees Starting degrees Starting 
irregular relation to each other from the Y-axis from the Y-axis 
round 
rectan ular power and information interface 
g typing the FE Fi 
bore hole definition interface |top positioning 
round — regular - hole pitch diameter Starting angle starting iti i 
number of holes diameter from the X-axis bottom positioning 


right positioning 


round - regular - positionin |positioninYof bore diameter of hole N 


number of holes X of hole N hole N left positioning 
rectangular - bore hole distance in hole distance in Y centered 
irregular - diameter X 


number of holes 


massive joint structure 
rectangular - position in positionin Yof | bore diameter of hole N z =a 
Irregular - XofholeN hole N pipe joint structure 
number of holes Fa: 
square-type tube joint structure 


Fig. 2. Construction instruction for adapter plate. 


Flange patterns can be categorised as round or rectangular. This is primarily determined 
by the parameter F. The next step is to define the boreholes, so these are first to be 
defined in the number (A) and their arrangement on the flange plate (N). Depending on 
the arrangement, the position and diameter of the holes must then be specified (B, T, 
A). Thus the flange patterns are defined holistically, whereby now the position and 
orientation of these to each other must be determined (X, Y, Z, RX, RY, RZ). In order 
to connect these flange patterns, the characteristics of the structure (S) and the course 
ofthe connecting structure in space (X, Y, L) must now be determined. This makes it 
possible to define the connection structure in such a way that interfering contours which 
the area of application of the robot entails are taken into account, thus extending the 
accessibility and versatility of use for parts which are difficult to access. Finally, the 
type (V) and positioning (P) ofthe required energy and information interfaces must be 
defined. A database with conventional power connectors and hydraulic connections is 
used for this purpose. When positioning, the user only defines the side of the position 
of the connections. The exact positioning and the required holes or grooves for fas- 
tening these holders or bushings are set automatically. Defined space requirements on 
all sides of the plugs or holders are defined by boarding boxes. 
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3.2 Generation of geometric flanges using constructive solid geometry 


In order to compute appropriate geometries, we introduce parametric geometries. 
For this, we define an XML syntax that allows the specification of constructive solid 
geometries. The syntax specifies three different kinds of objects: solids, mathematical 
operations, and loops. 

Solids are defined by a region within the x/y-plane and an extrude operation. By 
now, we have two operations implemented. The first one extrudes the specified region 
linearly in z-direction. The second one revolves the region around the x-axis. In addi- 
tion, any solid can be moved and rotated arbitrary in space. 

Based on the solids, we can create more complex geometries by using mathematical 
operations. Merge operations are used to combine two or more solids to a single one. 
Subtract operations will remove one or more solid from another one. 

Loops can repeat both solid creation and mathematical operations with a given num- 
ber of times. An accumulating transformation specifies a relative offset for each inter- 
action step. 


<?xml version="1.0"?> 
<ParametricXML xmlns:xsi=http://www.w3.org/2001/XMLSchema-instance 
xmlns:xsd="http://www.w3.org/2001/XMLSchema"> 
<Parameters> 
<Parameter Name="Radius" Symbol="R" Value="100"/> 
<Parameter Name="HolePositionRadius" Symbol="R1" Value="80"/> 
<Parameter Name="NumberOfHoles" Symbol="N" Value="6"/> 
</Parameters> 
<Difference> 
<Color Value="#008800" /> 
<Extrude Length="20"> 


<Path Startx="-{R}" StartY="0"> 
<ArcTo X="{R}" Y="0" CenterX="0" CenterY="0" ClockWise="false" /> 
<ArcTo X="-{R}" Y="0" CenterX="0" CenterY="0" ClockWise="false" /> 
</Path> 
</Extrude> 


<Loop Count="{N}"> 
<Displacement RZ="360/{N}" /> 
<Union> 
<Extrude Length="10"> 
<Transform X="{R1}" /> 
<Path Startx="-5" StartY="0"> 
<ArcTo X="5" Y="0" CenterX="0" CenterY="0" ClockWise="false" /> 
<ArcTo X="-5" Y="0" CenterX="0" CenterY="0" ClockWise="false" /> 
</Path> 
</Extrude> 
<Extrude Length="10"> 
<Transform X="{R1}" Z="10" /> 
<Path StartX="-10" StartY="0"> 
<ArcTo X="10" Y="0" CenterX="0" CenterY="0" ClockWise="false" /> 
<ArcTo X="-10" Y="0" CenterX="0" CenterY="0" ClockWise="false" /> 
</Path> 
</Extrude> 
</Union> 
</Loop> 
</Difference> 
</ParametricXML> 


Fig. 3. XML example for a parametric flange. 
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Fig. 4. Image of the geometry specified by the XML document depicted in Fig. 3. 


For all three kinds of objects, parameters can be specified. Hence, we can define a 
parameter-based template for various geometries. By changing the parameters, we can 
influence the shape in a simple manner. Of course, there are constrains in the construc- 
tion possibilities. The drawing plane for new regions is always the x/y-plane, for in- 
stance. However, the specification is made with respect to simplify the parsing and 
computation. 

Fig. 3 shows an example for a simple flange. A web-service accepts the document 
and computes the geometry. We have implemented various export options, e.g. STEP, 
JSON geometry format, and 3D view embedded within an HTML document. Fig. 4 
shows the computed geometry. 


Fig. 5. Image of a complex adapter plate geometry 
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With the web-service, besides such relatively simple geometries, almost all possible 
geometric forms are conceivable. The potential of the service is demonstrated by the 
exemplarily generated geometry in Fig. 5. This adapter plate should connect a gripper 
with a rectangular flange and a robot with a round flange. This flange, with its extension 
and angulation, serves to keep the relatively thick flange geometry of the robot further 
away from the poorly accessible handling position. This means that greater accessibility 
can be achieved even in confined spaces. The parametrisation happened via the XML 
schema described above. 

In the future, the parameterization will be extended by a user interface, so that the XML 
schema is generated in the background. The aim is to guide the client through a user 
centred configuration process that is as simple as possible. 

In addition to the generation of adapter plates, the web service is so generically struc- 
tured that it can also be used for the generation of various geometries in the sense of a 
micro service architecture. Overall it is currently used to generate simple workpieces 
on the ROBOTOP platform[9-10]. 

Furthermore, optimization potential was seen in the geometry of the adapter plates of 
low stressed areas, which are to be generated in the future on the basis of the finite 
element structure synthesis. 


4 Results 


In conclusion, an early automated generation of adapter plates reduces the engineering 
effort and enables the consideration of this adapter plate already in the early planning 
phase of a robot system. In this context, the dead weight and the extension of the ac- 
cessibility of the robot can play an important role in the design of the system. 

An automated generation of such adapter plates was implemented as an example. 

In the case of automated generation, a generally valid description of these plates and 
thus also of the flange images is necessary. At present, however, such standardisation 
is lacking in the areas of robot flanges, gripper flanges and adapter plates. A generally 
valid design specification for robot adapter plates was developed in the context of this 
paper. For the generation of the CAD geometry, an automated generation was realised 
on the basis of the developed construction regulation, medium Constructive Solid Ge- 
ometry. Here the developed web-service is fed by means of an XML based syntax and 
allows an almost unlimited free space in the geometry design. 

In this context we were able to prove, that the developed design rule for the definition 
of an extended adapter plate is working well. The generation via CSG turned out to be 
practicable. 

The parameterisation of the adapter plate is to be extended in the future, by a specially 
developed user-centred interface and then integrated into the ROBOTOP platform. 
Momentarily, the optimisation of the adapter plate geometries still poses a difficulty, 
here a finite element structure synthesis could be carried out with the aim of reducing 
the weight of the adapter plate. 
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Abstract. A reconfigurable production is able to adapt to a new prod- 
uct in a short time. This can be built up from cyber-physical production 
systems (CPPS) which exchange various data with several devices simul- 
taneously via a convergent network. In these future production systems, 
convergent networks will connect more participants than real-time pro- 
duction networks do today. In order to enable uninterrupted operation, 
reconfigurations must not affect existing communication. Due to this lim- 
itation, fragmentation of the configuration can occur, restricting future 
operations. This paper investigates fragmentation caused by reconfigu- 
rations using a simulation to determine whether fragmentation can lead 
to problems in future production networks. 


Keywords: real-time production network, fragmentation, reconfigura- 
tion 


1 Introduction 


The industry 4.0 production must be able to produce a wide range of variants 
up to batch size one and at the same time execute the production in the shortest 
possible time [1]. In order to achieve this, [1] and [15] propose a division into 
cyber-physical production systems (CPPS) consisting of a set of sensors and 
actuators integrated into a network. In the context of assembly, this corresponds 
to a switch from inflexible conveyor belts to automated guided vehicles (AGVs) 
[2]. CPPSs enable a reconfigurable manufacturing system that produces different 
products and can be adapted within a shorter period [16] instead of a flexible 
manufacturing system that can produce different variants of a product. 

This concept does no longer fit into the automation pyramid, with clearly 
divided field and control level for each manufacturing system [4, 11]. Real-time 
communication occurs only at the lowest levels in locally limited real-time pro- 
duction networks (RTPNs) (e.g. within a machine), whereby sensors and actua- 
tors communicate with a single controller. The concept of levels dissolves during 
the transition to a CPPS-based production environment. The communication 
across levels increases so that, for example, an AGV independently accepts the 
next available production order. In parallel, the values of sensors are not only 
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used for control, but are also stored and evaluated by applications, for exam- 
ple in order to improve the early detection of errors. Instead of separation into 
levels, this results in a network of different applications that communicate with 
each other. Real-time communication is no longer limited to local networks and 
spans the entire network from CPPS instead, as shown in Fig. 1 [15]. 


Enterprise 


O Real-time 
resource planning network 


Plant level 


-time 
communi- 
ation 


Process 

control level 
Control ASTARA a 
level 
Field 
level / 


Real-time 
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Fig.1: The automation pyramid dissolves and separated RTPNs turn into one 
large RTPN. Adapted from [15]. 


Two possible network architectures that realize a CPPS-based automation 
are explained in [7] and [8]. Both topologies integrate all participants into one 
RTPN, which can be implemented with the vendor-independent time-sensitive 
networking (TSN) extensions to the IEEE 802.1 standard, for example [10]. The 
control is done in local cloud platforms, which execute applications for control. [8] 
also explains that this control can be executed on external cloud platforms. The 
network architecture is shown in Fig. 2. 
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Fig. 2: Network architectures in CPPS-based production systems. 


The main difference to the previous architecture of different, isolated RTPNs 
is that all network participants are contained in a shared RTPN. This results 
in a RTPN with more participants than before, which can span an entire shop 
floor. Due to the local separation of the sensors/actuators and the controller, 
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the data must be transmitted over several switches, which must be configured 
in such a way that all real-time critical data reach their destination on time. 

Due to its increased number of devices and new flexibility, changes may 
occur more frequently to this RTPN that require a new configuration of the 
switches. Examples for changes are the addition or removal of devices from the 
network. Furthermore, when starting a new application, such as real-time anal- 
ysis of manufacturing data, additional data may need to be exchanged between 
the sensors/actuators and the execution platform. This must not affect the cur- 
rent communication. So, the rest of the production is not interrupted by the 
reconfiguration. In contrast, [9] only takes care during reconfiguration so that 
as few packets as possible fail and a modification of existing communication is 
tolerated. 

The paper is structured as follows: In Section 2, RTPNs with minimum guar- 
anteed latencies are introduced, followed by a description of the reconfiguration 
of these networks without interruption of the existing communication and the 
possibility of fragmentation in Section 3. In Section 4, a simulation of fragmenta- 
tion is done and evaluated. Afterwards, a summary is given and possible further 
work is explained in Section 5 and Section 6. 


2 Real-time Ethernet 


RTPNs require minimal latency and jitter, which can be achieved by transmis- 
sion at specified time slots. In addition, the CPPS architecture requires support 
for multiple senders and receivers. This can be accomplished, for example, by 
using TSN with 802.1Qbv [6]. It is assumed that all participants have a syn- 
chronized time, for example through the Precision Time Protocol (PTP) [5]. 
Within TSN, data are classified into traffic classes. The real-time capability is 
realized by defining that only data of a certain traffic class are transmitted in a 
time slot. Data of other traffic classes are not transmitted in this time slot. The 
communication runs cyclically, whereby the synchronized clocks ensure that all 
participants use the same synchronized cycle. This cycle is divided into time slots 
and provides an exclusive transmission path for the data of a traffic class. The 
division can be defined per port at the switch and does not have to be identical 
for the entire network. 

In Fig. 3 this distribution is visualized in an example. A cycle from t to t+1 
is shown. From t to t+¢t, only data of the traffic class 1 can be sent, from t+t, to 
t+tz only traffic class 2 can be sent. The same applies to the third slot. Finally, 
there is a slot from t + t3 to t + 1 which allows to send all data. This cycle is 
repeated continuously. 

In the following, a common cycle time is assumed for the entire network. In 
case of different transmission cycles a hypercycle with cycle time of the least 
common multiple of all cycle times can be considered equivalent [3]. Also, only 
the traffic sent in a defined time slot is considered as real-time traffic in this 
article. Prioritized traffic, which is sent only prioritized to other traffic, is not 
considered separately and is seen as part of non-real-time traffic. 
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Fig. 3: A cyclic transmission with reserved time slots (1 to 3) guarantees that 
only traffic of specific class is transmitted within this period. 


3 Reconfiguration of real-time production networks 


As mentioned in Section 1, due to the larger RTPN and new possibilities to 
start/stop applications that access real-time data across the network, changes in 
the configuration of the RTPN are more likely. A reconfiguration adds and/or 
removes any number of time slots for real-time data. Existing time slots for 
real-time data cannot be modified (e.g. moved or split), as otherwise guarantees 
regarding latency and jitter could be violated [12]. This section points out which 
conditions must be met to reconfigure a network without disturbance to the 
existing real-time traffic. Afterwards, the problem of fragmentation is addressed 
as a consequence of the reconfiguration. 


3.1 Execution of a reconfiguration in a real-time production 
network 


When executing a reconfiguration that is not intended to restrict the function- 
ality of the entire network, the possibilities of reconfiguration are limited in 
contrast to a fresh configuration without existing time slots. Time slots with- 
out real-time data do not have any requirements. These can be moved at will, 
shortened by time slots with real-time data (inserting a time slot at the edge) 
or interrupted (inserting a time slot in the middle). 

In general, changing the configuration can be done in two steps. First, all 
time slots that are no longer required are removed from the current configuration. 
Afterwards, the newly added time slots are inserted into the available time ranges 
in the cycle that are not yet occupied with real-time data. Possible incremental 
algorithms that support the placement of new time slots are, for example, [12] 
and [13]. 


3.2 Fragmentation 


When removing and inserting time slots of different sizes, gaps may occur be- 
tween time slots because existing time slots remain in the cycle at their previous 
time during reconfiguration. Within the context of RTPNs we define fragmenta- 
tion in a communication cycle when the reserved time slots have gaps between 
each other so that the remaining free space in the communication cycle is not 
available as a single block. This can cause problems when adding a larger time 
slot and there is no sufficiently large gap available even though the total sum of 
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time slots is less than the cycle time. This is similar to external fragmentation 
of storage systems, when files have been removed and added over time causing 
separated blocks of free memory [14]. However, the strategies to reduce external 
fragmentation (e.g. compaction) on storage systems can not be applied to frag- 
mentation in RTPNs, as moving existing time slots within a cycle may violate 
guaranteed latencies. Fragmentation in the context of RTPNs is not the same as 
IP fragmentation, when IP packages are splitted during transmission. 

An example of fragmentation is shown in Fig. 4, which visualizes a sequence 
of reconfigurations. At the beginning (step 1), there are five time slots, which 
occupy the whole cycle. Then, the time slots 1 and 4 are removed and the smaller 
slots 6 and 7 are inserted at their position (step 3). After removing the time slot 
3, there are gaps between all existing time slots in the resulting cycle so that 
time slot 8 cannot be inserted although there is enough space in total. 
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Fig. 4: Example for fragmentation during reconfiguration. The time slot 8 can 
not be scheduled, although the total space is enough. 


This demonstrates how fragmentation can cause problems over time, as small 
gaps between time slots cannot be used for larger time slots. To what extent and 
after how many reconfigurations this problem occurs is evaluated in the following 
using a simulation. 


4 Simulation 


In the simulation, fragmentation is evaluated based on a real-time connection 
between two network participants by removing and inserting multiple time slots 
of different sizes. It is not considered that the time slot in a network must be 
reserved on the complete path from the sender to the receiver. Fig. 5 shows this 
conflict. A time slot for data from switch A over B to C must be available in 
both A— Band B + C. On the A > B route and on the B — C route there 
is enough space. However, on A — B the new time slot can only be inserted in 
the middle, while on B — C it can only be inserted at the end. Consequently, 
there is no time for the new slot that is available on both A > B and B > C. 
Since a solution for scheduling in networks is NP-hard [12] and different 
algorithms produce different scheduling results, only a single connection is con- 
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Fig. 5: A time slot across multiple switches requires free space on each connection 
at the same time. (Transmission time expected to be zero) 


sidered in the simulation (e.g. the A > B connection). With this simplification, 
the results are expected to be better than in a network with multiple switches. 

The simulation considers a cycle that is reconfigured several times. For the 
simulation a 100 Mbit network equivalent to current field buses is assumed, 
which exchanges data with a frequency of 1 kHz. Thereby, 12500 Bytes can 
be transmitted per cycle including the overhead of the protocol (e.g. Ethernet 
header). Below is a list of all the parameters of the simulation, some of which 
have been chosen identically to enable a comparison: 


— Number of time slots (X) 

— Number of time slots that are removed and inserted each iteration (Y) 
— Minimum and maximum size of time slots 

— Maximum number of iteration. Here: 10000 

— Total size of the cycle. Here: 12500 Bytes 


The simulated reconfiguration sequence is shown in Algorithm 1. Each time a 
new time slot is added, its size is randomly determined from a parameterized in- 
terval. During initialization, X time slots are inserted into a cycle whose size for 
this simulation is set to 12500Bytes. The time slots are all inserted into the cycle 
one after another so that there are no gaps between them (as shown in Fig. 5 in 
B- C with four time slots). Afterwards, the following steps are repeated: First, 
Y random time slots are removed. Then, Y random time slots are inserted into 
the existing gaps again. If several gaps are large enough for a time slot, the small- 
est gap is selected in order to have larger gaps available for any following, larger 
time slots. The two steps of removing and adding time slots are performed until 
either the time slots can no longer be inserted or the maximum number of itera- 
tions has been reached. Up to 10000 iterations are executed in this contribution. 


Insert X time slots next to each other; 
for i = 1 to 10000 do 
Remove Y random time slots; 
Try to insert Y random time slots with random sizes, each time slot 
uses the smallest available gap; 
if Insert failed then 
| break; 


Algorithm 1: Simulation of reconfigurations. 
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For each parameter set (P) listed in Table 1, 1000 runs with different random 
sized time slots are performed. Each run contains up to 10000 reconfiguration 
iterations. The first parameter set (Pı) of the simulation uses 40 time slots (X) 
of a size of [249 : 250] and replaces one time slot per round (Y) so that the 
interval of 12500 is occupied between 79.68 % and 80.32 % in total. The result is 
shown as a histogram in Fig. 6. The histogram uses a bucket size of 100 so that 
the blue bar on the left represents the 133 simulations that failed within the first 
100 iterations. The red line represents the total percentage of simulations that 
failed until an iteration, e.g. at 10000 it has a value of 93.5 %. Thus, 935 from 
1000 runs of Pı failed before iteration 10000. 

As a result, only 65 runs out of 1000 were capable to process 10000 itera- 
tions, while 25 % of the runs were not able to reach 200 iterations. Transferred 
to production, this means that in case of this parameterization, in 25 % of the 
cases no 200 reconfigurations of the network can be performed without the re- 
quirement of moving previous scheduled time slots. But depending on the initial 
configuration and on the required changes, it might also be possible to reach 
over 10000 reconfigurations. 
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Fig. 6: Histogram of 1000 runs of parameter set P}. 


Table 1 lists all simulations performed with parameters and results. All simu- 
lations have in common that they use 80 % of the available cycle (1000 of 12500) 
in average. The first column is the number of the parameter set. The next three 
columns contain the parameters used in that simulation, the number of removed 
and inserted time slots per iteration (Y), the range of time slot sizes and the 
total number of time slots (X). 

The other columns show evaluated results of the 1000 runs per simulation. 
”Min./max. iterations” is the minimum and maximum number of iterations. A 
value of > 10000 means that the simulation has stopped after 10000 iterations. 
”Failed” contains the count of the 1000 runs in total that do not reach the 10000 
iterations, ” passed” contains the runs that reach 10000 iterations. The last two 
columns show the iteration when 25 %/90 % of the runs have failed. When less 
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than 25 %/90 % of the simulation runs failed in total, there is no value (”—”). In 
Fig. 6 this value is the number of iterations when the red line reaches 25 %/90 %. 


Table 1: Parameters and results of the simulation 


Y |time slot sizes} X ||min. iterations|max. iterations|failed|passed|25 %|90 % 
P, |1 249, 251 40 13 > 10000 935 | 65 | 172 |7900 
P|1 248, 252 40 16 > 10000 998 2 195 |2847 
P |1 247,253 40 19 7200 1000; 0 154 | 1986 
P,|l 246, 254 40 21 5434 1000| 0 148 | 1633 
P; |1 245, 255 40 22 4059 1000| 0 123 |1414 
Ps |1 124, 126 80 128 > 10000 11 | 989 - - 
Pz |1 [84, 86] 118 339 > 10000 1 999 - - 
Ps |1 499, 501 20 8 1465 1000; 0 33 | 290 
Py |2 249, 251 40 14 > 10000 965 | 35 | 152 |6888 
Pıo|3 249, 251 40 8 > 10000 953 | 47 | 293 |6756 
Pıı|4 249, 251 40 6 > 10000 950 | 50 | 528 |7804 
Pı2|5 249, 251 40 6 > 10000 930 | 70 | 662 |8752 
Pi3|6 249, 251 40 5 > 10000 904 | 96 | 882 |9896 


Regarding the simulations P, to Ps, the maximum number of iterations de- 
creases from > 10000 to 4059 when the size of time slots varies more ([249, 251] 
to [245, 255]). So, wider range of time slot sizes can therefore reduce the number 
of successful reconfigurations. This can be explained by the strategy that the 
smallest possible gap is used when inserting a time slot. When the largest time 
slot should be inserted, in Pı, only two other time slot sizes ([249, 250]) might 
occupy a previous removed largest time slot, while in P; ten smaller time slots 
([245, 254]) can occupy a largest time slot. Thus, the chance of finding a match- 
ing gap between two time slots is the smaller the more possible time slot sizes 
are available. 

Using smaller but more time slots in Pg and Pz, compared to Pı, fewer runs 
(11 in Pg and 1 in Pz) failed within 10000 iterations. One explanation for this is 
that more iterations are needed to fill 20 % of the available space if no smaller 
gap is found for an inserted time slot, as the time slot is smaller. By doubling the 
size of the time slots (P compared to Pı), the maximum number of iterations 
decreases from > 10000 to 1465. 

Comparing Pı with P 9 to Pig, the amount of passed runs varies within a 
range of [35,96]. In Pio to Pig the number of iterations till 25 % percent of 
the runs fail is higher than P,. So, increasing the number of changed time slots 
per reconfiguration (Y) from one to two firstly causes more failures. From 3 on 
(Pio to P13) more reconfigurations are possible before 25 % of the runs failed. 
Two effects might cause this behavior. At first, the first iteration may remove 
only time slots of the minimum size and only inserts slots with a greater size 
so that all inserted slots must be placed at the end of the previously inserted 
time slots. When this is done for more slots simultaneously, then more gaps are 
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in the cycle and the buffer at the end becomes smaller more quickly. Otherwise, 
the more time slots are reconfigured, the more likely it is that the size of some 
removed time slots will match that of some time slots to be inserted. In case of 
40 reconfigured time slots per iteration, all time slots are removed and inserted 
again, so there will be no fragmentation at all. 

In summary, the simulation of a single wire between two switches identifies 
the following behaviors: 


— The greater the amount of sizes, the lower the average number of successful 
reconfigurations. 

— Smaller time slots enable more reconfigurations than larger time slots. 

— The reconfiguration of multiple time slots at once has a positive effect on 
the number of reconfigurations, after a deterioration at the beginning. 


5 Summary 


In a reconfigurable production based on CPPS, new network architectures with 
larger RTPNs are used. The increased number of participants and the common 
reconfiguration of the production lead to an increased number of reconfigura- 
tions of the RTPN in operation, whereby parts of the configuration must be 
exchanged without affecting the existing communication. If real-time communi- 
cation is implemented as a cyclic data transport, in which fixed time slots are 
reserved (e.g. using IEEE802.1Qbv), the reconfiguration cannot move existing 
time slots. This can lead to fragmentation when time slots are added and deleted 
during reconfiguration. Within this paper a simulation has been used to check 
how often a network can be reconfigured for specific sets of parameters. Depend- 
ing on the parameters, a RTPN can only be reconfigured a few hundred times 
before gaps between existing time slots prevent a successful reconfiguration. 


6 Further work 


On hard disks, the problem of fragmentation has been solved by running defrag- 
mentation operations that move existing files on the disk. To solve the problem 
of fragmentation in RTPNs equivalent concepts need to be developed and meth- 
ods for modifying existing time slots during a reconfiguration are required. One 
solution is that some applications can be customized to tolerate scheduling the 
time slot to a new position during runtime within a certain range. In future 
publications we will try to move existing time slots repetitively only a tiny bit 
(e.g. only several nanoseconds) over time. This should not interrupt the existing 
communication and enables drifting of existing time slots to reduce gaps. When 
this is done during reconfiguration larger gaps for inserted time slots should be 
achieved in the progress. 
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Abstract. The growing complexity of industrial plants, enormous ad- 
vances in autonomous driving and an increasingly digitized world re- 
quire a quickly rising amount of virtual planning and testing using three- 
dimensional simulations. In addition to the number of technical advances, 
the complexity of individual objects and scenarios is increasing, pushing 
the requirements on simulation technology to generate sufficient realistic 
data within limited time frames. 

Simulation frameworks have their means to store the information nec- 
essary to represent the properties of all scenario entities — the simula- 
tion state - in their database. However, they lack simulation algorithm- 
dependent transformations and augmentations of the simulation state 
that allow maximizing calculation efficiency. This paper introduces a 
generalized formalism to describe these transformations and illustrates 
how they help to bring the simulation framework closer to a special- 
ized simulation database that stores spatial information with a focus on 
performance - the spatial database. 

Further, this paper demonstrates the advantages of the approach with the 
introduction of two concrete transformations. One targets the efficient 
representation of three-dimensional spatial relations for industrial robots 
while the other allows generating different levels of complexity in the 
definition of materials in the context of autonomous driving. 


Keywords: spatial database, simulation state transformation, high per- 
formance, digital twins, three-dimensional scenarios 


1 Introduction 


Simulation of large scale, three-dimensional scenarios has become a major de- 
mand of Industry 4.0. The introduction of Digital Twins storing complex ge- 
ometric and dynamic data requires specialized databases, able to store such 
information efficiently. Furthermore, scenarios as Hardware in the Loop (HIL), 
including sensors as LIDARs, cameras or other spatial sensors, require sufficient 
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realistic data while maintaining real-time capabilities. To meet both needs, a 
combination of ray tracing and rasterization algorithms must be used. Another 
field which demands efficient spatial data structures are dynamic simulations. 
Since the geometry approximations in the dynamic simulation have significant in- 
fluence on precision, complex geometry approximations are required for realistic 
dynamic simulations. To handle collision detection for such complex geometries, 
efficient collision queries are a precondition to enable interactive modeling. Fur- 
thermore, increasingly specialized hardware is developed to accelerate various 
tasks in the domain of ray tracing, rasterization, general vector operations and 
machine learning tasks. To exploit these devices, the input data must be trans- 
formed into a device-specific form. This paper introduces a generalized approach 
to satisfy the proposed constraints on real-time performance by exploiting re- 
cent hardware acceleration technologies and efficient data structures leading to 
a spatial database. To reach this aim, a database transformation formalism is 
introduced and applied to two challenges of spatial simulation. 


2 Related Work and Theory 


In this section the theoretical bases for this work are explained. Beginning with 
the idea of spatial database, continuing with a formalism to describe simulation 
systems and models and finally introducing a formalism to describe simulation 
state transformations. 

The idea of spatial databases is to transform or augment the information in 
a database - the simulation state - with respect to hardware capabilities, per- 
formance constraints and requirements on precision in a way that either three- 
dimensional simulation algorithms can process them more efficiently or to enable 
them in the first place by providing a specialized representation of the informa- 
tion. 

In general, a simulation has the purpose to create an environment, in which 
a system with all its dynamic processes can be analyzed as an experimentable 
model in order to reach findings which are transferable to reality [10]. Moreover, 
since simulations can be arbitrarily complex, it is quite difficult to find a com- 
mon ground or a superordinate system to describe them. One approach proposed 
by [11] is the formalism of Discrete Event System Specification (DEVS). DEVS 
can be applied to implement behaviors defined by basic system formalisms such 
as discrete time and differential equations. The DEVS structure is defined by 


M= (U, S, Y, Sint» dext; As ta) (1) 
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where 
U is the set of input values 
S is a set of states 
Y is the set of output values 
dint: SS> S is the internal transition function 
dext:Qx X >S is the external transition function, where 
Q = {(s,e)|s € S,0 < e < ta(s)} is the total state set 
e is the time elapsed since last transition 
A:$-Y is the output function 
ta: S > Rio is the time advance function 


The proposed system M is in a state s € S at any given time. In case no external 
event occurs, the system remains in its current state for a period ta(s). At the 
end of the period ta(s), the system output value is evaluated by A(s) and the 
state changes to Öjn+ (5). For the case that an external event u € U occurs during 
ta(s), the system changes its state to ext (U, €, s). 

In order to introduce a concept of time to the DEVS, [11] defines time as t = 
(T, <) where T is a set and < is an ordering relation of elements of T with < 
transitive, irreflexive and antisymmetric [11, p. 128]. Furthermore, the trajectory 
w:(tı,ta) > A with A an arbitrary set, describes the motion through the set A 
beginning at tı and ending at ta for every t € (t1, t2). The motion at time t is 
defined by w(t). The second formalism in [11] describes an input/output system 
as 


Sio = (T, U, 2, Y, Q, A,A) (2) 
where 
T is a base time 
Q is a set - the set of allowable input segments 
Q is a set, the set of states 
A:Qx2>Q is the global state transition function 


A:QxU—>/Y(orA:Q-—Y) is the output function 


The essential functions are A(q,w) and A(q,z) with q E€ Q, u E€ U andw E€ R, 
which describe the state transition function and the output function respectively. 
In this definition, w(t) is the input function. Figure 1 visualizes the system Sio. 


Fig. 1: Input /output system 


The introduced input/output system represents arbitrary simulation functions in 
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a generalized form. In a simulation framework, both functions map to algorithms 
implemented in software or specialized hardware, leading to a system or model 
specific behavior. However, depending on available hardware, requirements on 
precision or aim for closeness to reality, the specialized forms A of A and A of A 
should be used to increase performance and to meet the required precision. The 
application of A and A also requires the adapted version ĝ of q. 

To transform q to ĝ this paper introduces the input transformation ĝa of the state 
transition function A and the input transformation 6, of the output function A 
as follows 


6A:Q7 Qa C QU Len, , the input transformation of the state transition function 
ôa: Q> Qa C QU Len, , the input transformation of the output function 


To augment the state q information leny about the simulation algorithm or 
the hardware executing the algorithm can be used to transform the internal 
representation. Transforming q can also mean to remove information irrelevant 
for the algorithm, while adding information relevant for the algorithm. Figure 2 
visualizes the augmented system. 


Fig. 2: Augmented input/output system 


The last element to complete the picture is the definition of simulation state 
q. For a simulation framework [7] proposes the generalized model state to be 
defined as 


q = (x,a, A) (3) 


with 
x € V"* model state 
a € Vie model parameters 
A € V"j* model algorithms 


Given the theoretical basis of the DEVS, its specialized form - the input/output 
system - the transformations ĝa and 6, and the definition of q, the next section 
illustrates with two examples how these transformations would look like aug- 
menting a simulation state. In both examples the aim is to increase performance 
and efficiency doing the special tasks of spatial collision queries and rendering 
materials. 
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3 State Transformations 


This section introduces two state transformations showing the superiority of us- 
ing transformed states ĝ in the system functions A and A. The state transforma- 
tion design is derived from the requirements which three-dimensional simulation 
scenarios including real-time, sensor simulation, rendering and dynamic simula- 
tion impose. 
The first simulation state augmentation aims on increasing the efficiency of spa- 
tial collision queries, ray tracing and occlusion queries. The common basis of 
these methods is that they need to determine which objects or geometries are in 
particular region of the scene. While ray tracing has a lot of application in sensor 
simulations for LIDARs or ultrasonic sensors, collision queries are a major part 
of rigid body simulations and occlusion culling in render algorithms. All together 
they play a primary role in the scenarios mentioned in section 1. 
Given the general definition of q from section 3, the model state x contains 
the information about the object geometries, poses, the physical properties such 
as mass, material, etc. The model algorithms A are among others: ray tracing 
algorithms, collision detection algorithms and occlusion culling algorithms. 
Since all algorithms require an efficient representation of spatial relations in 
the scene, the first proposed augmentation of q adds Space Partitioning Struc- 
tures (SPS) to the model state x, leading to the augmented state dsps. da.sps 
and 6,,spg are in this case the same function, containing an algorithm which 
creates the SPS from the geometry and pose information in x. An SPS divides 
the three-dimensional space into a hierarchical structure of geometry groups, 
which are close to each other as shown in figure 3 left. This allows highly ef- 
ficient broad intersection queries due to the hierarchic structures, similar to a 
tree as shown on the right of figure 3, which allows skipping a vast amount of 
geometry intersection tests. 
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Fig. 3: Two-dimensional Bounding Volume Hierarchy 


While there exist many SPS such as Binary Space Partitioning (BSP) [8], Oc- 
trees [1] and Bounding Volume Hierarchies (BVH) [3], not all support efficient 
building algorithms making use of multi-core systems, which is required when 
dealing with dynamic scenarios. A very efficient BVH construction algorithm 
was developed by [12]. It allows to build a BVH in linear time, in parallel on 
CPU or GPU and is therefore be used for further implementation. 
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BVH recursively enclose larger groups of geometries in axis aligned bounding 
boxes, beginning with a single geometry and ending with one root box, enclos- 
ing the entire scene. Figure 4a and 4b illustrate how the BVH encloses the objects 
more and more precisely as the tree depth increases. To trace a ray through the 
BVH, the ray is initially tested against the root box. On intersection, the tree 
structure is tested recursively, using the depth first approach. The two individual 
child nodes of each node are traversed in order of the nearest intersection. To 
find broad intersections between geometries, the geometry to test is enclosed in 
a box, which is recursively tested against the BVH beginning with the root. A 
leaf node intersection while testing against the BVH gives a potential collider, 
which needs to be tested using a narrow collision algorithm. Using an SPS many 
of broad range problems can be solved efficiently. 


(a) rasterized (b) ray-traced 


Fig. 4: BVH enclosing scene objects 


Figure 5 shows how the BVH can be used to accelerate collision tests in the 
example of a reconfigurable working cell [6], by successively test for collision 
against smaller groups of objects. The Digital Twins of the robots execute the 
task of picking and placing work pieces into specialized machines. Those process 
the work pieces and the robots move the processed items to the target carriage. 
To determine whether the robot can pick a work piece an intersection query 
has to be performed. This test is split into a broad and a narrow phase. Before 
preforming the narrow collision test, the broad hierarchical test is performed, 
testing against each level of BVH from top to bottom as explained before. 

The next proposed augmentation of q considers A and x, targeting the materi- 
als used to describe the properties of the objects in the scenario. The material 
description has influence on how sensors perceive objects and how objects are 
visualized. This is especially important for simulated cameras and LIDARs. How- 
ever, while these sensors require very complex material definitions, others, like 
contact sensors, only required very basic material information. This requires a 
general material definition from which different levels of complexity can be de- 
rived using the output transformation 64, 7p. Since öa,mpr depends on the 
hardware used for rendering, Jen, contains information about its raster and ray 
tracing capabilities. For each derivation, the rendering functions expressed in 
shaders and contained in A have to be adapted depending on the complexity 
of the material in x and env. One material definition, which allows this deriva- 
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(d) BVH level 3 (e) BVH level 4 (f) BVH level 5 


Fig. 5: BVH in reconfigurable working cell 


tion, is the Material Definition Language (MDL) by NVIDIA [2]. MDL allows 
defining arbitrarily complex materials and deriving material models with fixed 
complexity depending on the users’ requirements. Figure 6 shows the usage of 
different material derivations - generated from a single material definition - in 
different render techniques. 


(a) real-time rastered (b) interactive rastered (c) ray tracing using [9] 


Fig. 6: Subset of the distilled static material model 


4 Implementation 


This section describes the implementation of ö4,upr using the MDL to trans- 
form the simulation state q depending on the available hardware. The transfor- 
mation is required to provide the necessary information for different rendering 
techniques such as ray tracing or rasterization, which represent the output func- 
tions A,; and Ara, respectively. The proposed implementation allows deriving the 
required material representations from a single material definition. In the chosen 
implementation, the rasterizer uses the OpenGL API [4] and the ray-tracer uses 
the Optix API [5]. Moreover, the implemented rasterization approach is able to 
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switch between procedural and baked material rendering (as shown at the end of 
section). The ray tracing is realized with Optix, which a fully customizable ray 
tracing framework using GPU hardware and software acceleration. Rasterization 
is implemented in GLSL using OpenGL as graphics API. 

The MDL material definition uses recursively-refined expressions to describe the 
surfaces, beginning with a basic layer consisting of a surface, volume, index of 
reflection, thin walled flag and a geometry information. Every surface expres- 
sion in the MDL can be a bidirectional scattering distribution function (BSDF) 
leading to a structure shown in figure 7 on the left. 


Quality Performance 
Ray Tracing Rasterization Rasterization 
Complex Material Definition Static Material Baked Material 
(params) ze Definition Definition 
[Params BSDF Distilling i >|_cituse_| 
[paar sr — ma ] 
paar BSDF 
[pars] cor 


Fig. 7: Different MDL complexity representations 


Since these BSDF can only be evaluated in a ray-tracer, the rasterizer requires 
a different representation. For the chosen implementation, the following expres- 
sions are evaluated during rasterization: Anisotropy, anisotropy rotation, atten- 
uation color, attenuation distance, base color, clearcoat normal, clearcoat rough- 
ness, clearcoat weight, metallic, normal, opacity, roughness, specular, subsurface 
color, transmission color, transparency, volume index of reflection. To retrieve 
the expressions, the NVIDIA MDL distilling API is used. The distiller generates 
the simplified expressions for the required static material model. 

The generated static material model can be used in two ways: The first is the run- 
time evaluation of the expressions as figure 7 middle shows, leading to materials 
with a resolution up to the precision of the underlying floating representation. 
Moreover, run-time evaluation enables the usage of procedural material, which 
use noise functions to generate permutations of different base material layers, 
resulting in a very realistic and nonrepetitive structures in the material. The 
second way is to evaluate the expressions within a fixed range and store the 
result in a texture as figure 7 right shows. This process is called baking and has 
the advantage that complex run-time calculation (depending on the material) 
are reduced to a simple texture lookup. The downside is that the baked textures 
have a limited maximum size, resulting in a fixed sampling frequency and region 
to be captured. This is especially problematic for procedural textures, since the 
baking removes their nonrepetitive structure. 

In summary, the simulation state transformation is implemented to derive three 
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material representations from a single definition, leading to a flexible, target 
device-specific material representation. 


5 Evaluation 


In this section, the performance of the output functions Art Ara introduced in 
section 4 is evaluated. Ara uses rasterization with either baked or run-time eval- 
uation of the static material model. The measurement consists of two scenarios. 
The first one is a test scene containing simple geometry with 12016 triangles, 
which use MDL materials. The second one is the urban test drive scenario shown 
in figure 6 with complex geometries containing 3085073 triangles. To evaluate 
the performance the rendering time of one frame At is measured. The measured 
results in table 1 indicate a big difference in render times between ray tracing 
and rasterization, which renders ray tracing impracticable on weaker GPU de- 
vices. The simulation state transformation is therefore the enabling step for the 
visualization on weaker devices. 


Renderer Method At minAt maxAt % of max triangles 


GLSL Baked raster 16 14 20 100 3085073 
GLSL Expr raster 33 28 36 206 3085073 
Optix ray tracing 164 150 190 1025 3085073 
GLSL Baked raster 4 3 4 100 12016 
GLSL Expr raster 5 4 6 125 12016 
Optix ray tracing 34 32 45 850 12016 


Table 1: Measured frame times in ms using NVIDIA GTX 1080 GPU 


6 Summary 


This paper introduces the idea of spatial databases as the transformation or aug- 
mentation of the information in a database in a way that three-dimensional sim- 
ulation algorithms can process them more efficiently creating a spatial database. 
Further the idea is expressed by extending the DEVS formalism by the simulation 
state transformations 6, and ĝ4. Using the formalism two state transformations 
aiming on increasing computation efficiency and on creating hardware depended 
representations of q are introduced. For the latter the implementation and eval- 
uation is described. The results gathered from the implementation of the MDL 
augmentation indicate the need for variable simulation state representations to 
support multiple levels of hardware compute capabilities. The next steps are to 
investigate in different levels of geometry representations such as signed distance 
field or voxelization to provide an even wider range of simulation state represen- 
tations, enabling more devices to use spatial databases in real-time applications. 
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Abstract. The Internet of Things (IoT) is a network between physical 
and virtual things, software services and people. The specific services 
these things offer to set up value-added chains are provided by the 
computer resources they use — like cloud computing for computationally 
intensive tasks or edge computing for real-time processing. Concepts 
such as the Digital Twin enable things to operate autonomously and 
connect independently. This leads to self-initiated systems which require 
the development of new infrastructure concepts to combine different 
computer resources and various Digital Twins. This paper proposes the 
Smart Systems Service Infrastructure (S'I) — a multi-domain networking 
infrastructure which allows things to choreograph themselves situation- 
specifically in highly decentralized and dynamic environments. 


Keywords: IoT Infrastructure, Digital Twin, Choreography, Value-Added 
Chains 


1 Introduction 


In the last decade the term Digital Twin emerged and several definitions were 
proposed. The glossary of platform Industry 4.0 defines Digital Twins as virtual 
digital representatives of physical assets [6]. More detailed perspectives focus on 
simulation or the networking of physical assets [2,5,14]. In [11] the definition is 
extended to a representation which includes semantics and is used as a structural 
element. Depending on the domain-specific development, the scope of operation 
offered by the Digital Twin varies. In addition to that, the question how Digital 
Twins can be used to set up decentralized value-added chains, where Digital 
Twins live, how they interact with software services and humans in a holistic 
approach is still being researched intensively. 


In this work the Digital Twin is considered to represent one asset in the IoT. 


Figure 1 shows the individual components of the IoT in an hierarchical order. At 
the lowest level are physical assets that are connected to their Digital Twins via 
sensor/actuator networks. The corresponding Digital Twins as well as software 
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Fig. 1. The Digital Twin in the IoT. Figure based on [10]. 


services and Man-Machine Systems (MMS) are shown in the IoT-levels above. 
These things — the Digital Twins representing assets, the software services and 
the MMS - are the nodes of the IoT. MMS provide Human-Machine Interfaces 
(HMI) which enable the operators to interact with the IoT-nodes in the specific 
field of application. 


For this, Digital Twins, software services and MMS offer various services 
according to their specific roles as nodes of the IoT. Applications are configured 
and tasks are carried out because the individual participants of the IoT network 
and collaborate with each other creating problem-specific configurations of IoT- 
nodes making up the “applications”. It is the purpose of all the things — and 
especially the Digital Twin — to connect with other things i.e. Digital Twins, 
software services and MMS to create such applications and perform certain tasks. 
This way, concrete applications are configured by selecting and networking the 
things provided by the IoT. 


All things — Digital Twins, software services and MMS - have different 
requirements on the computer resources they use. For a machine which has 
to perceive and react to its environment with the help of a sensor network, a 
reaction in real-time is only possible if the processing takes place on the machine 
itself. However, space and energy resources might be limited. Therefore resource- 
intensive calculations that do not have real-time requirements are outsourced to 
calculation services which are reached via the IoT. This illustrates that only the 
combination of different computing resources enables the collective execution of 
a task. 
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The sensor-example above describes one thing’s interaction with one software 
service. The example can be expanded to a whole network of things where a 
multitude of Digital Twins, software services and MMS connect with each other, 
thus creating applications solving different tasks. Individual things provide their 
functionality to diverse applications, resulting in a dynamic network which adapts 
situation-specifically to particular tasks. Orchestration in which the process logic 
is implemented by a sequence control that calls up the individual processes is no 
longer possible. Things act autonomously and pursue various goals independently 
of each other. The orchestration becomes a choreography in which the individual 
things contain the process logic. 

Summing it up, an infrastructure is needed that allows the things to chore- 
ograph themselves using the computer resources they need. The work at hand 
presents such an infrastructure which we will call S3I in the following. For that 
purpose, in the following section, the state of the art is pictured. In Section 3 
the requirements for and the modules of the infrastructure are presented. An 
example application is given in Section 4. We conclude the paper in Section 5. 


2 State of the art 


There is a wide range of publications on the individual computing concepts (e.g. 
cloud computing and edge computing) [1,8, 13,15]. Approaches that combine the 
various computer resources, ranging from the cloud to edge devices, in one single 
infrastructure are rare, though. 

Alam et al. [2] propose a combination of cloud and edge computing. They use 
the cloud for the Digital Twins to ensure the scalability of storage, computation, 
and cross domain communication capabilities. The physical layer which represents 
the edge of the network provides real-time sensory fusion. Two things can establish 
peer-to-peer (P2P) connections either through direct physical communications or 
through indirect cloud-based Digital Twin connections. The combination of a 
fuzzy rule base with a Bayesian belief network forms a decision algorithm that 
selects between the physical-physical interaction, the cyber-cyber interaction and 
a hybrid way which uses both, the physical and the cloud-based communication. 

In addition to scientific approaches there are industrial approaches. The 
Bosch IoT Things Platform suggests an IoT Gateway Software which provides 
autonomy and intelligence at the edge [4]. The gateway software supports current 
connectivity protocols so that the IoT devices can connect to the gateway and 
benefit from its computing resource. On the other side the gateway offers a local 
abstraction and allows unified access to the connected devices and their data. 
Cloud Computing is used as a back end to handle tasks with advanced computing 
requirements. 

A combination of cloud, fog and edge computing is done by Bierzynski 
et al. [3]. Using improved IoT hardware, edge and fog things can handle not 
only functional tasks but data processing and analyzing tasks like filtering and 
determining correlations as well. Therefore, the aim is to integrate the edge and 
fog things and to distribute the workload optimally on all levels (cloud, fog, edge). 
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For the distribution of tasks machine learning is proposed. As an example the 
light control of a fair hall is used. Within that example things are orchestrated 
by a central unit with no need for self-choreography. This approach does not 
answer the question how the infrastructure looks like supporting heterogeneous 
things which act autonomously and choreograph themselves. 

All in all, from the authors point of view, concepts to combine different 
computing platforms exit in principle but there are no infrastructural concepts 
which allow autonomous things distributed over the different areas of the IoT 
(cloud, edge, etc.) to choreograph themselves. 


3 Smart Systems Service Infrastructure (S?I) 


To allow highly decentralized things which use various computing resources to 
choreograph themselves there are several requirements for the infrastructure these 
things are using. 


1. The infrastructure shall allow the things to find each other. By finding 
each other the things can identify the participants within the infrastructure 
enabling the first step to choreograph themselves. 

2. The infrastructure shall allow the individual things to interact. The aim is 
that the things first find each other and then communicate with each other 
in such a way as to enable them to choreograph themselves. 

3. The infrastructure shall support a dynamic configuration of things. This 
requirement allows the things to act flexible and to react dynamically to 
changes within the network. 

4. The infrastructure shall provide security for the participants and their data. 
This includes protection against malicious attacks by third parties as well as 
confidentiality and integrity of data. 

5. The infrastructure should offer a means of communication. Therefore the 
participants do not have to implement their own communication infrastructure 
and might use the provided one instead. 

6. The infrastructure should have low overhead and should not demand extensive 
computing resources. This way, the infrastructure can be put together using 
edge devices only. 


The SI answers these requirements with different modules (Figure 2), which 
are explained in more detail below. 


3.1 Directory 


The directory is the infrastructure’s core. It provides a central metadata collection 
for the network’s things, which can be searched for, created, modified and deleted 
according to the client-server principle. 

To ensure confidentiality of this data the directory has to provide authentica- 
tion and authorization of the thing or human-being (called user in the following) 
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Fig. 2. Structure of the Smart Systems Service Infrastructure. IdP = Identity Provider. 


accessing the directory. The authorization part uses password or token authenti- 
cation to assure that the user is the one he claims to be. Once this is verified, 
the directory gives access to users on the basis of their respective roles. Access 
can be restricted or granted down to individual directory entries and is divided 
into read and write permissions. 

In addition to the confidentiality requirement, the directory also enables 
interaction between the things. Therefore, the directory does not only collect 
the things data but also provides their communication interfaces via so-called 
endpoints. The endpoints represent the interfaces of the things through which 
they make data and services available for other things. To be communication 
protocol-agnostic, the directory offers all data which is needed to access these 
interfaces. For example, this additional data might include a topic which is used 
within the MQTT protocol. 

Apart from the protocol-specific information on the endpoints, information 
on the data behind the endpoint has to be available. This information indicates 
the structure of the data and the format in which it is presented and using 
the data model introduced in the following. Additionally the directory enables 
the endpoints to administrate data structure and data format themselves. That 
way, dynamically changing data models within the things as well as dynamically 
changing thing configurations can be accommodated for. This also allows the 
introduction of undisclosed things. 


Data model Figure 3 shows the conceptual data model for the S?I directory. 
The root represents the entry point to the data and comprises the managed 
entities and things. Entities are users or groups of users that own, manage or 
otherwise relate to one or more things of the IoT. For the sake of clarity, user 
and group details are omitted. Things can be associated with a location to allow 
for spatial queries (e.g. find all nearby assets in a certain radius). Furthermore, 
each thing can be associated with communication endpoints for access through 
connectivity protocols like OPC UA, MQTT, HTTP or S°I broker. Things might 
also be hierarchically structured with other (child) things that are separately 
managed within the S?I directory. For example, a machine thing might consist of 
several machine child things like an engine or a chassis. Finally, the structure of a 


132 


- apiVersion: string 


0..* 


entities things 


ThingAssociation 
| 


argetThing 


Entity 
- identifier: UUID - 
- name: string 


/\ 


0..* 


dataModel: string [0..1] 
identifier: UUID 

location: Location [0..1] 
name: string 


values 


endpoints 


children allEndpoints 


1:.* $ 
Psi 


member Endpoint 
type: string 0 O- attribute: string 
uri: URI k - value: string [0..*] 


subgroup 


Fig. 3. Data model of the S?I directory. 


thing can be disclosed in more detail within the directory. For that purpose, the 
utilized data model can be specified. In more detail, (parts of) the thing structure 
can explicitly be exposed using a generic object model built from objects, values 
and links between objects. This allows to associate endpoints directly to the 
object or value they provide access to. This simple object model serves as a meta 
model on which most data models can be mapped thus supporting arbitrary 
application domains in the S®I directory. Providing only meta data results in low 
overhead and reduced computational requirements, so that the infrastructure 
can be set up using low-power devices only. 


Implementation From a technical point of view, the directory is a database 
that provides an API and takes care of authorization. This set-up can be provided 
by open-source tools like Eclipse Ditto. Eclipse Ditto is an open-source framework 
for Digital Twins in the IoT. Ditto focuses on solving the responsibilities a typical 
“back end” has in IoT solutions [7]. 


To store the data, Eclipse Ditto uses a MongoDB. Besides the data of the 
directory entries, Ditto also stores user information to authenticate and authorize 
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the user. The modeling of the users as entities is shown in Figure 3 and mentioned 
above. The authentication of the users is done by user credentials or JSON Web 
Token (JWT) which correspond to a digital ID card. Authorization takes place 
via so-called policies, which are assigned to the corresponding directory entries of 
the things. A policy lists the specific roles and defines the resources the respective 
roles can access. The resources can be whole entries of the directory or parts 
within the respective entry. For every resource read or write permission can be 
granted or revoked. Using a JWT, Ditto authenticates the user and separates 
its role (e.g. owner or administrator) from the JWT. In combination with the 
policies, the access rights to a requested entry are described. 

Ditto provides a REST API which offers the possibility to configure the 
policies as well as the specific entries. The computing resources on which the 
directory is based must therefore have [P-capable interfaces. Besides that, the 
directory can be set up anywhere, no matter if it is a cloud or an edge device. 


3.2 Identity Provider 


While authorization in the directory is done using these policies, for the authen- 
tication the user (i.e. a thing or an human-being) has to specify his data with 
user credentials or a token. 

Using user credentials, thus password authentication, the directory would 
have to store the user credentials of each user. On the other side the user has to 
have the corresponding password for the directory ready each time another thing 
is invoked or some other call is made. In a network of services and Digital Twins 
where a lot of access points are used this is not convenient. Also, the user data, 
as very confidential information, would be sent along every time a request is sent. 

For these reasons the S°I provides a central Identity Provider (IAP). The IdP 
manages the user credentials like passwords and certificates. Each user has an 
identity within the IdP, which is valid for other services at the same time. The 
identity is carried in form of a digital ID card, which is implemented as a JWT, 
as explained above. This digital ID card is the access key for the various things 
within the S3I and authenticates the user. 

The login of the respective users is done with OpenID connect, which pro- 
vides an authentication layer on top of the OAuth 2.0 authorization framework. 
OpenID connect completes the OAuth 2.0 authorization flow with authentication 
mechanisms [9]. 


3.3 SI Broker 


With the Directory and the Identity Provider participants are given the ability to 
make their interfaces securely accessible to others. The actual data transmission 
is handled by the participants themselves. To provide one message-based commu- 
nication option the S?I offers an Advanced Message Queuing Protocol (AMQP) 
broker. 

AMQP supports a request/response and publish/subscribe architecture. In 
cooperation with the messaging broker the protocol ensures a robust data transfer 
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and allows asynchronous communication by enabling messages to be stored in 
queues. Within the broker an exchange receives the messages and routes the data 
into the correct queue. The queue, in which the message belongs, is determined 
by a binding. The exchange also offers broadcasting of messages and topic-specific 
forwarding. Although the exchange types and the bindings offer a variety of 
communication architectures there can be custom exchange types like geodata- 
base routing. 


4 Application 


One field of application is the forestry and timber industry. It offers a multitude 
of different stakeholders and a heterogeneous structure [12]. In addition to forest 
rangers, forest workers and forest owners, facilities such as sawmills and forest 
machine manufacturers operate in the cluster. Regarding machinery there are 
forestry machines like harvesters, forwarders and skidders as well as hand-operated 
machines like motor saws and brushcutters. Furthermore there are services for 
forest growth simulations and tree species classification. 

The machines might have real-time applications running on the machines 
themselves thus representing edge computing. Services like forest growth simula- 
tion might run in the cloud using the computing power provided here. 

These things build a highly decentralized network with heterogeneous com- 
puting resources. Still they act independently and choreograph themselves to 
perform solutions in constant exchange. 

One example to illustrate the core concepts of the S®I is the communication 
between a sawmill and a harvester. The sawmill determines its need for a specific 
amount and type of wood. The aim of the sawmill is to place this order with a 
harvester. Optimally, an order to a harvester which is currently in a suitable forest 
area and can process the order is created. The sawmill logs into the directory and 
queries the harvesters within a specific area using a location based search. The 
directory entry which is shown in listing 1 provides information on the endpoints 
offering the current work status of the respective harvester. After choosing a 
suitable harvester the sawmill queries the directory entry of the harvester and 
uses the denoted endpoint to place its work order. 

This example illustrates how the directory is used. Using the REST API, 
things can query the directory and search explicitly or categorized for things. 
Each thing can indicate its interfaces declaring endpoints in the directory which 
is reduced to an AMQP endpoint in the example. Giving information about the 
exchange and queue which is used in the example, a communication partner is 
provided with the necessary information about the interface. 

With the login at the Identity Provider, the user gets an ID token to au- 
thenticate at the Directory, which allows access to the information about the 
AMQP endpoint of the harvester. This assumes that the harvester has listed 
the sawmill in the directory as an authorized party before. Since the harvester 
controls its policies (e.g. using the API of the Directory), it determines the users 
of its interfaces by itself. 
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1 { "name": "HarvesterJD_OPS", 

2 "identifier": "tzccokgc-resc-glpodnchfgtq", 

3 "location": { 

4 "lat": 51.454602, 

5 "long": 8.006526 

6 is 

y Topjacests | 

8 { "class": "workOrder", 

9 "identifier": "tzccokgc-resc-rpkmxnwurzgd", 
10 "endpoints": [ 

11 { "type": "sssiBroker", 

12 "uri": "“amqp://sssi/forestAuthority", 
ig "queue": "harvesterJD_OPS", 

14 "exchange": "sssi" 

15 } 

16 ] 

17 } 

18 ] 

19 } 


Listing 1: Directory entry of the harvester. 


5 Conclusion 


This paper presents an infrastructure for highly decentralized things. Since 
these things require diverse computer resources and connect autonomously and 
situation-specifically to applications, an infrastructure is needed which supports 
this choreography and which allows to use the computer resources meeting best the 
requirements of each individual thing. The requirements are specified and further 
the modules of such an infrastructure are presented. To allow situation-specifically 
choreography a directory is proposed, which offers thing- and user-specific access 
to the information needed for the thing to interact. More precisely, the things 
describe their interfaces and indicate which other things are allowed to access to 
the respective interfaces. 

Since the directory is reachable via a REST API, the things are offered a 
convenient way to manage their interfaces and the access to them. This way, the 
participants are independent of the computing resource they are using. Offering 
gateway services, things without IP-based communication can be included. 

With the Identity Provider a single entity is given which issues identity tokens 
to provide a comfortable authentication at the services, user interfaces and Digital 
Twins within the network. 

Further work will determine the computing resources that the S3I uses. Cloud 
computing might be convenient for applications where no other computer resource 
is available that is accessible from all over the respective network. Still, there 
might be applications that prefer computing resources at the edge level, dealing 
with privacy issues. 
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Abstract. Despite the far-reaching technological developments in the field of 
robotics (e.g. lightweight robots), the degree of implementation of innovative so- 
lutions in industrial practice is still comparatively low. This applies in 
particular to inexperienced users from small and medium-sized companies 
(SME). To facilitate the transfer of robot applications into practice, an internet- 
based configurator has been developed within the ROBOTOP research project. 
The configurator comprises planning functions along the whole engineering 
process. With regard to these functions, industrially implemented best practices 
are a main focus within the concept phase. Furthermore, they form a foundation 
for all further planning steps. In this context, the term ‘best practice’ refers to 
successfully implemented industrial applications. The acquisition, structuring 
and integration of these best practices into the web platform constitute the core 
of this paper. In order to highlight the emerging considerable benefits for indus- 
trial end-users, the digital implementation of best practices within the framework 
of the ROBOTOP internet platform is presented in the last step. The continuous 
use of digital planning platforms can ultimately pave the way for a significant 
increase in the degree of implementation of innovative robot applications. 


Keywords: Best practices, data acquisition, classification, robot configuration, 
Industry 4.0 


1 Introduction 


In the field of robotics, extensive research results have emerged in recent years. Despite 
the considerable potential of new technical achievements, e.g. lightweight robots, these 
systems often have not yet found their way into industrial practice. One reason for this 
is that the planning and configuration process requires industrial users to have extensive 
prior knowledge [1]. 

In order to counteract this problem, a modular and open internet platform with 
engineering functions along all phases of automation planning is developed in the 
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ROBOTOP research project. The aim is to use industrially implemented best practices 
as a foundation for a need-based planning process [2, 3]. On the basis of this planning 
process, inexperienced industrial end-users are introduced to the field of robotics. In 
this way, the mass market for industrial and lightweight robots will be opened up in the 
future [4, 5]. 


2 Planning Approaches 


In order to automate a production process by means of industrial or lightweight robots, 
various planning steps are necessary. In this context, numerous planning approaches 
were developed within the last years. These can be divided into application-neutral 
approaches [6], assembly-related methods [7] and automation-related systems [8]. 

Application-neutral planning methods are characterized by a more general 
description of the planning contents which makes them suitable for a wide range of 
applications. The REFA planning method was developed in the form of a guideline and 
can be used by industrial users for work design in general [9]. The method is suitable 
for various application scenarios that can relate to production as well as to the areas of 
service and administration. However, the main weakness of application-neutral meth- 
ods lies in the lack of a connection to the field of robotics, e.g. configuration of robot 
system and safety requirements. 

Assembly-related approaches are particularly suitable for detailed planning of work 
and assembly systems. In this context, Hartel and Lotter [10] developed a method for 
the planning of production lines from the assembly area. A special feature of this 
method is its high flexibility, as the individual product and production parameters of 
the examined application scenario can be varied, e.g. product design, product 
complexity and sales volume. Another example is the reference planning process [11] 
which can be used to plan various industrial scenarios, such as production, logistics, 
assembly or layout planning. Moreover, the Bosch planning guideline is based on lean 
production and scalable automation concepts and supports industrial users in all phases 
of production planning [12]. Overall, the assembly-related approaches have many 
advantages. However, the area of robot-based automation is not sufficiently considered. 

Automation-related approaches serve to design work sequences that are carried out 
by humans or robot systems. A large part of these procedures is based ona skill-oriented 
allocation of work tasks. Here, the consideration of different levels of automation 
[13, 14], concept-based assembly planning [15-17] and MABA-MABA diagrams 
[8, 18] should be mentioned. A central criticism of these approaches is that they only 
consider a very specific part of the overall planning process and, for example, do not 
provide approaches for detailed and implementation planning. 

In total, there is a large number of different planning approaches available which, 
however, have specific disadvantages with regard to the underlying planning task. In 
particular, the usability for industrial users without specific prior knowledge in the field 
of robotics is very limited. On this basis, the ROBOTOP configuration process for the 
implementation of robot applications has been developed. Here, inexperienced end- 
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users benefit from digital assistance tools and best practice solutions so that SMEs in 
particular gain access to robotics. 


3 Internet-based Configuration Process 


The aim of the internet-based configuration process is to facilitate planning procedures 
for both robot suppliers and customers and thus link both groups together directly. As 
indicated in Fig. 1, the ROBOTOP platform offers extensive assistance functions in all 
phases of automation planning. 


assistance functionalities along the entire engineering tool chain 


Fig. 1. ROBOTOP configuration process 


In the course of an initial selection of best practices, potential end-users first have 
the opportunity to describe their individual application scenarios (e. g. manual work- 
stations) using a structured and user-friendly questionnaire. Suitable best practices are 
derived in an automated selection process. 

The best practices serve as an initial concept proposal for the end-user and as a basis 
for the following individual configuration process. In this course, suitable robot sys- 
tems, peripheral components and gripping systems can be freely combined with each 
other, whereby the compatibility of the individual components is guaranteed at all 
times. 

To increase user-friendliness, the assembled systems can be directly transferred to 
an online 3D simulation. Users can make an initial assessment of the technical 
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feasibility of the assembled solution for their specific application scenarios. Depending 
on the result, it is then possible to go through upstream process steps again to improve 
the planning results. Alternatively, potential service providers may be contacted 
directly to obtain quotations. However, in order to achieve such high quality results in 
later planning phases (e. g. individual configuration, online 3D simulation), the 
selection of suitable best practices at the beginning is indispensable. 


3.1 Fields of Application for Best Practice Acquisition 


To ensure that best practices can be used for as many planning scenarios as possible, 
they should represent ab road cross-section of industrial applications. Therefore, 
frequently occurring industrial tasks were determined and categorized. 

Within the framework of a literature research, the results of various studies on the 
frequency of specific types of robot applications in industry were consolidated [19]. In 
addition, concepts and guidelines for the description of work tasks for robot systems 
were examined [20, 21]. The result was a heterogeneous overview of different robot 
operations. 

This overview was revised by experts from industry and research in various surveys. 
As a result, nine elementary robot applications could be identified. As indicated in 
Fig. 2, these could be divided into four higher-level categories. 


Fig. 2. Best practice classification 
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The common feature of machine loading / unloading, pick and place activities as 
well as positioning tasks is that individual objects are picked up, moved and placed in 
a different location. For this reason, the operations are assigned to the "handling" 
category. Furthermore, composing an object places higher demands on a potential robot 
system, since one object has to be inserted specifically into another. Similar to the 
screwing of objects, these activities increasingly occur in industrial assembly. For this 
reason, assembly and screwing are assigned to this category. Further typical tasks of a 
robot system are scanning of features and quality inspection. Since optical sensor sys- 
tems are often used for this purpose, these activities are assigned to the 
"optical inspection" category. In addition to the handling and inspection of 
components, robots are increasingly used in industry for coating and machining 
processes (in particular cutting). These activities have been classified in the 
"machining" category. 

Overall, robot systems are used in industry for a variety of other application 
scenarios (e. g. welding). However, a central requirement of the web platform is to 
comprise representative best practices that allow inexperienced users in SME an easy 
introduction to the field of robotics. This can be well ensured by the described best 
practices classification. 


3.2 Best Practice Database 


Based on the developed structure, industrial application partners within and 
outside the ROBOTOP research network were approached for the collection of best 
practices. This includes SME as well as large corporations in the automotive industry, 
machine and plant construction, packaging industry and air conditioning technology. 
This way, best practices data was collected for all four categories. However, an above- 
average number of handling tasks occurred. These are scenarios that can be automated 
easily and mainly without the aid of complex sensor technology. This underlines the 
current industrial trend that lightweight robots are used increasingly for simple appli- 
cations. In addition, there is the significant advantage that these 
applications offer an ideal planning guideline for inexperienced users from SME. 

In order to select suitable best practices in the first step of the configuration process, 
it is necessary to compare the requirements of the examined user scenario with the 
properties of the best practices. Therefore, the best practices are described thoroughly 
on the basis of defined characteristics. This was ensured by the development of a 
uniform data sheet. 

The best practice attributes include metadata of the individual components such as 
robots, grippers, tools and conveyors, types of material feeding and safety-related 
equipment. Moreover, the IP protection class, component properties, control systems 
as well as the upstream and downstream process are taken into account. On the basis of 
the generated data, ROBOTOP can provide industry-independent best practice 
solutions for various application scenarios. In this way, the project can significantly 
promote knowledge transfer to SME. 
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3.3 Selection of Best Practices 


For the selection of best practices as a planning foundation, some information on the 
specific investigation scenario, e. g. manual assembly workstation, is required. This 
data is collected using a structured questionnaire. This includes questions relating to 
the work task to be examined and questions relating to the handling objects. 

The required information on the work task mainly refers to the four categories 
presented in this paper (i.e. handling, assembly, optical inspection, machining). This 
allows eliminating unsuitable best practices that do not correspond to the field of 
investigation. 

As anext step, information on the handling objects is required since these have 
significant influence on the automation capability [22]. The queried object data includes 
the geometric body (ellipsoid, cuboid and cylinder) as well as information on the 
maximum dimensions. Furthermore, data regarding component stiffness, material 
properties and component mass is collected. Based on this product data, further 
improper best practices can be eliminated. For example, the weight of the handling 
objects has significant influence on the payload and thus on the size and type of the 
entire robot system. If the weight range of the application scenario does not correspond 
to the best practice, the best practice is evaluated as unsuitable and excluded from the 
configuration process. In this way, it is possible to carry out a meaningful pre-selection 
and to propose suitable best practices to industrial users. 


4 Application 


The planning logic as well as the best practice solutions were integrated into the 
ROBOTOP internet platform as part of a 3D demonstrator. The digital representation 
contains all CAD models necessary for the visualization of the best practices. This 
includes material feeding components, the representation of products, load carriers and 
conveyor elements, the robot system with its gripper or tool components, safety engi- 
neering as well as the switch cabinet. By means of this level of detail, the ROBOTOP 
configurator is provided with sufficient information about the correct functionality and 
visualization of the production process. 

Based on the requirements analysis, suitable best practices are pre-selected for the 
user in the first step. As it can be seen in Fig. 3, industrial end-users can view suitable 
best practices in a 3D environment and compare the essential properties with each other. 
For this reason, the configuration interface displays the corresponding attributes. This 
includes information on the kinematics used, the end effector and the estimated price. 
Furthermore, the platform includes a rendering of the scene to get an initial overview 
of the components used. This allows industrial users to select individually one of these 
suitable best practices for further configuration steps. 

Furthermore, the user has the possibility to customize his robot application in the 
configuration mode. Various components such as workpieces, feeds or safety features 
are available for selection. Using the component-positioning menu, suitable elements 
can be selected and added to the overall system. 
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Best Prach 


BP 01: Loading a machine wih sheets Load Machine UR-5 sucöon gripper (in-house producion) 
BP 02: Loading a packaging machine Pick And Place UR-5 suction gripper (in-house producäon) 
BP 03: Screw connecson to a Screw UR-3 screwdriver (in-house production) 


loudspeaker module 


( BP 04: Exchange ofhokding mandrels Load Machine RSO10N PGN-Plus 
BP 05: Loading a grinding machine Load Machine Sawyer Sawyer standard gripper 
ERENTO Serw LBR iwa-7R800 KUKA LBR-AB-Class screwdriver 
( BP 21: Joining a brush holder Assembly YuMi - IRB 14000 IRB 14000 - gripper 
a 
© | . BP 22: Assembly and pressing Pick And Place YuMi - IRB 14000 sucäon gripper (in-house producson) 
h 


Fig. 3. Preselection of best practices on the ROBOTOP web platform 


This includes working tables, conveyors, robots and end effectors. In this way, the 
user can select between various manufacturers and choose the right object type. Align- 
ment can take place within all six degrees of spatial freedom. To ensure compatibility 
between, the configurator displays only suitable solutions (see Fig. 4). 


e e 1-1 1-1 


Fig. 4. Individual configuration process based on a suitable best practice 
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As can be seen in Figure 5, it is possible to store realistic data for each added object. 
In this case, the configurator menu shows the parameters for a new workpiece. At this, 
information concerning basic geometry, dimensions, weight and material can be edited. 
Due to the 3D environment and clearly defined input queries, no in-depth knowledge 
of robotics is required, so that almost every industrial end-user can benefit from the 
web platform. 


Applicaton Area 


Workpiece 


Metal (mag nec) 


Fig. 5. Editing of realistic workpiece data 


5 Conclusion and outlook 


Industrial robot applications must meet significant technical requirements to ensure 
high availability, quality and safety. In many cases, this results in highly complex 
applications. The successful planning of these robot systems requires a correspondingly 
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high level of specialist knowledge, a wealth of experience and extensive knowledge of 
the standardization situation. Especially for SME, this represents an entry barrier to the 
use of industrial robot applications. 

This problem was a main motivation for the ROBOTOP research project. The web 
platform enables industrial end-users to go through the entire configuration process 
from initial concept planning to preparation for implementation. By using industrial 
best practices as a planning foundation, users gain a practical insight into the field of 
robotics. In addition, the extensive assistance functionalities of the platform and the 
detailed 3D representation enable even inexperienced users to run through the entire 
planning process without any additional help. 

Overall, powerful web platforms mapping the complete engineering process are a 
key technology of the future. These can ultimately pave the way for a significant 
increase in the degree of implementation of innovative robot applications. 
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Abstract. In the wake of demographic change and increasing knowledge of po- 
tentially harmful activities and work areas, technical support systems like exo- 
skeletons are used to minimize the risk of musculoskeletal disorders (MSDs) by 
supporting employees during tasks with high biomechanical load. In addition, the 
increasing quality requirement and individualized demand make production ac- 
tivities more complex and changeable. Not least because ofthe well-trained abil- 
ities and skills of humans, employees have a central role in current and future 
value chains. One possibility for physically supporting manual tasks is the exo- 
skeleton. During the development and implementation ofthe exoskeletons, many 
of their effects and performances have been evaluated. Mostly, developers or 
end-users focus on selected biomechanical effects like reduction of musculoskel- 
etal stress. The effects on cognitive structures as well as the impact on work qual- 
ity are less known. The current study analyzed the cognitive effects during the 
use ofthe active exoskeleton Lucy for tasks performed at or above head level in 
a real environment (training center for scaffolders). The study was conducted 
with 18 participants. Our experiment was able to show that participants who were 
supported by the exoskeleton Lucy made significantly fewer mistakes in a work 
task (53,8%). In addition, a significantly higher cognitive performance (concen- 
tration achievement, processing of target objects and error frequencies) was 
found in supported participants, compared to the unsupported condition, in a psy- 
chometric test after each work task. 
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1 Introduction 


Despite the increasing demand in automation production tasks, humans have a cen- 
tral role in common, future value chains, and organizational structure [1]. The reason 
can be seen in the unique skills and abilities of the human perception, cognition and 
motor skills which are still superior to automated solutions in many areas, such as flex- 
ibility, sensorimotor skills, cognition and learning abilities, and attentive perception. 
On the other hand, we still have other areas, where automated solutions are advanta- 
geous, e.g. high endurance and accuracy, arithmetic operations or memory capacity. 

To support human beings during manual tasks, different approaches are possible, to 
offer different types of support. They can strengthen the position or substitute the em- 
ployee [2]. In general, such support technologies can be arranged parallel or serial to 
the user and support the user in physical and cognitive aspects, e.g., [3-5]. In this paper, 
we address an exoskeleton, which can be classified as a parallel system. Such systems 
are characterized by a force path parallel to the human musculoskeletal system. The use 
of technical support systems in the field of, e.g., production, logistics, and the handcraft 
may be motivated in different ways. Central drivers are the organizations or the em- 
ployee. Support systems can be used to increase quality and productivity or decrease 
musculoskeletal stress and thus the potential for musculoskeletal disorders. 

In general, exoskeletons are designed to achieve task-individual properties. In re- 
spect to the desired properties, such systems are different in kinematic structure, hu- 
man-machine-interfaces and actuation, as well as (in the case of active systems) sensors 
and control units [5]. Depending on the basic concept of each exoskeleton, such systems 
can support different kinds of tasks. Typical applications of exoskeletons in industrial 
areas are supporting tasks at head level or above, handling loads, gripping activities or 
handling tools and components [6-12]. 

The benefit of a mechanical relief for the workforce seems most obvious. Not least 
because the mechanical strain caused by the nature of the work task, such as working 
at head level or above, or handling heavy loads, could lead to fatigue and chronic com- 
plaints in the workforce (musculoskeletal diseases) [13]. These negative consequences 
must be avoided at all costs in order to be able to fulfill the central requirement of the 
work task: to be able to work without any impairment or damage [13, 14]. 

At this point, it should be emphasized that working at or above head level is an un- 
favorable working position for people, but is justified due to the requirements of pro- 
duction and cannot be changed in its actual execution. This may lead to the develop- 
ment of pain, which can also become chronic in the future [15]. A reduction in cognitive 
performance, which may be reflected, for example, in impaired memory function is 
directly related to chronic overload as the result of permanent physical strain. In partic- 
ular, these consequences are the reason for the development of technical support sys- 
tems for industrial production [16]. 

The performance of exoskeletons and their ability to reduce physical stress during 
manual activities has been shown in previous research [7, 9-12]. However, their influ- 
ence on worker’s cognitive performance and working quality are less known. The rela- 
tionship between the levels of perceived pain and reduced cognitive and physical per- 
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formance have been discussed extensively in psychological literature [17]. The inves- 
tigation of the impact of physical support of an exoskeleton on specifically cognitive 
performance a rather new field in both psychology and production engineering. Psy- 
chometric testing is necessary for the collection of objective data measuring cognitive 
performance. Here, the relationship between the test and the experimental objective 
was considered to allow a valid quantitative evaluation. Thus, in addition to measuring 
work task duration and therein occurring errors, the latest version of the d2-R Attention 
and Concentration Test was chosen as the central psychometric measuring device for 
data collection, which meets the above requirements. The abbreviation d2-R refers to 
the target item “d” with two bars that is intermixed with d, p and b with different amount 
of bars above and/or under the letters (“R” stands for revised). According to the manual 
of the d2-R Attention and Concentration Test, this psychometric test presents as valid 
and reliable. According to the TBS TK review (Testbeurteilungssystem des Testkurato- 
riums der Féderation Deutscher Psychologenvereinigungen), the d2-R Attention and 
Concentration is suitable for the evaluation of levels of concentration. Thus, all central 
criteria regarding concentration level following the work task could be recorded [18]. 

This paper reports a further study, focusing on the investigation of the influence of 
support systems on human cognitive function in a work place setting. For this purpose, 
participants performed a task with and without the support system. Afterwards a psy- 
chometric test procedure was carried out. A detailed description of the study design, 
including task setups, observed variables, statistical analysis and results can be found 
below. 


2 Study design 


The aim of this study was to analyze the effects of a physical support system on 
human cognitive function, especially the concentration levels, the treatment of target 
objects and corresponding error frequency in a psychometric test, and performance dur- 
ing a working task. In a previous publication it has been shown, that the exoskeleton 
Lucy successfully reduced physical stress [8] and improved the participants’ levels of 
concentration [19]. To investigate whether the physical relief effect of Lucy may lead 
to improvements in participants’ cognitive abilities within a workplace setting, testing 
was performed under realistic conditions in a training center for scaffolders. 

Key hypotheses were: 

e The physical support of an exoskeleton decreases the performance errors in the 
d2-R test, and in turn increases concentration levels and the number of pro- 
cessed target objects within a considered time unit of the d2-R test. 

e The physical support from exoskeletons decreases the occurrence of errors in 
the work task performed. 


2.1 Exoskeleton Lucy 


The exoskeleton Lucy [7] was used, which is illustrated in Figure 2. It is worn like 
a backpack and supports the user during movements at head level and above. It uses 
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pneumatic actuators to partially absorb and dissipate the additional workload, caused 
by and dependent on the tasks performed. In order to realize a natural level of support, 
predefined force curves are implemented in the control system. A detailed technical 
description can be found in previous work [7]. The user is able to change the level of 
support manually and individually applicable to each task. For standardization of the 
tests, all participants used the same level of support parameters. Hereby the level of 
support was set to approximately 4 kg with a shoulder angle of 90°. 


tool 


sensor N, 
actuator ~ 
arm rest 
harness system 


Figure 2. Exoskeleton Lucy for support tasks at head level and/or above. 


2.2 Experimental setup 


To investigate the hypotheses, 18 male participants performed a given task in a 
standardized manner. Participants were all second-year trainees within the scaffolding 
construction industry. Due to the absence of female trainees, it was not possible to ac- 
quire a gender-balanced sample for the survey-period. Mean age was 22.55 years (SD 
= 5.79; range: 17-39 years). Participants’ height ranged from 170 to 190 cm. 

In order to determine the interventional effects of the exoskeleton, participants were 
randomly allocated to one of two interventional groups. Participants with odd partici- 
pant number were assigned in Group 1 and with even participant number in Group 2. 
Group 1 began the work task with the Lucy system and then performed the same task 
in an unsupported condition. Participants of Group 2, began in the unsupported and 
then ended in the supported condition. Participants were instructed to take a break of 
10 minutes between the two conditions, respectively. During this time, participants 
were able to recover from the previous work task as well as from the concentration test. 
Thereby equal starting points were created for each experimental condition. For the d2- 
R concentration test a total of 14 consecutive letter rows consisting of "d" and "p" were 
shown to the participants in a pen and paper test. The task was to find and cross out the 
letters "d", which were marked with two dashes in the standardized interval of 20 sec- 
onds for each line. After the acoustic signal "Stop! Next line!" was given, participants 
had to move on to the next line following the same procedure as previously. This pro- 
cedure is in accordance with the manual of d2-R and took overall 280 seconds. To 
assess the effects of the intervention on concentration, participants performed another 
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d2-R concentration test in both conditions immediately after completing the work task 
(see Table 1.). 


Table 1. Experimental setup. 


begin test procedure end 
r work work 
test group sample baseline assignment 2 : pause | assignment 72 A 
size concentration Ist pass concentration 2nd pass concentration 


group 1: 
odd 
number 
participants 


group 2: 
evenumber 
participants 


A standard scaffold was erected, as it is in real life settings (i.e. facades). For the 
experiment, eight crossbars including type-related fuses and a standard carpenter's ham- 
mer, for hammering in the fuses, were used. A complete crossbar weighed 3.5 kilo- 
grams and was 1 meter long. All participants were familiar with the materials and meth- 
ods used in the work task. The work task was designed after an everyday occurrence 
within the scaffolding industry, consisting of inserting eight crossbars onto the highest 
level of the scaffold, hammering two fuses into each crossbar. The installation of one 
crossbar contained three steps (see Figure 1): 

1. receiving the crossbar from a colleague on the floor below 

2. positioning the crossbar on the highest installation level (2 meters from the run- 

ning board) 

3. fixing the crossbar by hammering in two fuses at either end of the board 


step 3 


Figure 1. Installation steps of a crossbar at the highest installation level which is 2 meters high 
from the running board. 
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During each installation, the crossbar was held at knee, hip, chest, and overhead 
level. The highest installation level was 2 meters from the running board (see Figure 
1). This was considered overhead level for all participants. The final installation step 
was perceived as the most physically demanding step by all participants, since both the 
crossbar and the hammer required excessive forces above head level. It was therefore 
chosen as the condition in which the errors during the working task (ErrorWT) were 
counted. Additionally, and more importantly for this study, Lucy showed to deliver the 
greatest level of support at overhead level. 


2.3 Observed variables 


To assess the impact on cognitive function after each run of the experimental condi- 
tions (supported/unsupported), measurements of concentration performance, accuracy 
in d2-R test processing (error frequency) and speed in test processing (detected targets) 
were obtained using the d2-R Attention and Concentration Test [20] as quantitative 
parameter. Additionally, the errors during the working task (ErrorWT) were defined as 
hits missing the fuse. 

Concentration performance (KL) was used as measure of concentration capability. 
In the test this described the number of crossed-out target objects, minus the number of 
wrongly marked objects. The processed target objects (BZO) were determined by the 
working speed. It was measured by the number of processed target objects of the d2R 
in a given test period. Error frequency (F) was defined as the accuracy during test pro- 
cessing and was indicated by correctly processed items. Therefore, high values in the 
error frequency meant better results than smaller ones. 


2.4 Statistical Analysis 


Four t-tests were conducted for this study. Three were used to analyze baseline data, 
determining whether there were significant differences between the two groups (sepa- 
rated for Working Order) prior to our experiment in the three variables of the d2-R. 
Another was used to see ifthe required working time for the same task under two con- 
ditions (supported, unsupported) differed from each other. To assess the effects of the 
Lucy for the two factors, the method Analysis of Variance (ANOVA) was used. An 
ANOVA was calculated for the three dependent variables of the d2-R (KL, BZO, F) 
each, and one for the errors occurring during the work task (ErrorWT). Each ANOVA 
had the same mixed within-between design with repeated measurement on the within- 
subjects factor Support (supported, unsupported) and the between-subjects factor 
Working Order (Group 1, Group 2). With the within-subjects factor the difference be- 
tween the two conditions with and without the support of the exoskeleton was esti- 
mated. The between-subjects factor indicated whether Working Order of supported and 
unsupported condition (Group 1, Group 2) had a significant effect. The interaction ef- 
fect of the two main factors could assess, if the effect of support was modulated by the 
Working Order. 

The experimental data were analyzed and evaluated using the software RStudio (R- 
3.6.1) (RStudio, 2012). For standardization of the d2-R values, norm tables printed in 
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the manual were used, considering the different age groups. The d2-R Attention and 
Concentration Test was standardized from the end of 2007 to mid-2008. The manual 
for the d2-R test did not provide more detailed information on the standardized period. 
Altogether 4024 persons with an almost balanced gender ratio (50.7% male, 49.4 fe- 
male) formed the basis for the standardization [20]. 


3 Results 


The analysis of the t-test regarding the d2-R baseline showed that there were no 
significant group differences as a function of Working Order for the concentration per- 
formance (KL) 416) = 1.81, p = .09, the processed target objects (BZO) 1(16) = 1.88, p 
= .084, as well as for the error frequency (F%) ¢(16) = 1.41, p = .18. The degrees of 
freedom (in brackets) of the t-test and ANOVA are an estimate of the number of inde- 
pendent information that went into calculating the estimated t-value. For the calculation 
of a between-group design the amount of groups (here 2) are subtracted from the sample 
size (18). The non-significant results of our analyses (p > .05) indicated that the null 
hypothesis could not be rejected and we could assume our groups had the same level 
of concentration prior to the experiment. The t-test of the time on task for conditions 
supported (mean = 150.61 s) and unsupported (mean = 132.17 s) differed significantly 
(16) = 2.13, p = .04). 

The results of the four within-subjects ANOVA’s for the variable Support (sup- 
ported, unsupported) are summarized in Table 2. An effect size (np2) of .01 is consid- 
ered as small, .06 as medium and .14 large [21]. All four dependent variables (KL, 
BZO, F and ErrorWT) showed significant main effects for the factor Support. Partici- 
pants in the supported condition achieved better values in the dependent variables of 
the d2-R and lower error counts in the work task, than participants in the unsupported 
condition. This suggests a positive influence of Lucy on the wearers. The main effect 
for Working Order did not reach statistical significance in any of the ANOVAs. 


Table 2. Summary of the main effects Support (supported, unsupported) of the Analysis 
of Variances (Support x Working Order). 


observed supported | unsup- F-value signifi- | mean effect 

variables (mean) ported F (1,16) cance squared | size 
(mean) value p | error np? 

MSE 

concentration per- 

formance (KL, d2-R) 103.28 98.28 65.45 <.0001 | 3.44 .80 

processed. target OUP | 1555 83 | 56.8 <.0001 | 3.05 

jects (BZO, d2-R) À 97. 6.87 ; : 78 

error frequency 

(F, d2-R) 104.56 100.33 12.25 .003 13.1 43 

errorscan ‘the work Ies 3.5 24.92 <.001 | 3.25 61 


task (ErrorWT) 
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The results of the interaction term of the four ANOVA’s between-subject factor 
Working Order (Group 1, Group 2) as a function of Support (supported, unsupported) 
are displayed in Table 3. The results show that Working Order had a significant inter- 
action effect on Support for the variables concentration performance and the processed 
target objects, but neither on the effect of support on the error frequency of the d2-R, 
nor on the errors in the work task. The significant interaction between Working Order 
and Support for the variables KL and BZO were due to the better values in the supported 
condition, when the participants had already executed the task in the unsupported con- 
dition (Group 2), compared to when participants started in the supported condition 
(Group 1, see Figure 3). 


Table 3. Summary of the interaction effects (supported, unsupported x Group 1, Group 
2) of the Analysis of Variances (Support x Working Order) 


observed variables F-value significance | mean squared effect size 
F (1,16) value p error MSE np? 
concentration 
performance (KL, d2-R) 23.56 .0002 3.44 .6 
processed target objects 
(BZO, d2-R) 20.13 .0004 3.05 .56 
error frequency 
(F, d2-R) 1.22 29 13.1 .07 
errors in the work task (Er- 
rorWT) 31 59 3.25 .02 
Assistance 
W supported 
| unsupported 


107 
106 
105 
104 
103 


ean value) 


m 
è 
8 


© 101 

100 
99 
98 
97 
96 


d2R - BZ 


Group 1 Group 2 


Group 1 Group 2 
(a) b 


d2R - F. (mean value) 
è 


Error Work Task (mean value) 


Group 1 Group 2 Group 1 Group 2 


(c) (d) 


Figure 3. Interaction plots of (a) concentration performance (KL), (b) processed target objects 
(BZO) and (c) error frequency (F) of the d2-R test, as well as (d) errors during work task (Er- 
rorWT). Group 1 had a working order of changing from supported to unsupported, while the 
Group 2 worked in a reversed order. 
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4 Discussion 


To evaluate the effects of a physical support system on cognitive function in a field 
setting (work place of scaffold trainees), the above study assessed participants’ perfor- 
mance in a psychometric concentration test (d2-R) and the errors made in the work task. 
The results of the study showed significantly better concentration performance, pro- 
cessed target objects, and the error frequency in the d2-R test as well as the errors in 
the work task, when participants were supported by the exoskeleton Lucy, compared to 
the unsupported condition. This may be an indication, that the decrease in physical 
strain through Lucy could improve cognitive performance of the user, compared to an 
unsupported condition. To rule out order-effects, the experiment was run in two meth- 
odological conditions (supported first, unsupported first) each with a subset of the sam- 
ple. The comparison of the standard values in the d2-R baseline measurement between 
the two groups did not reach statistical significance for each variable of the d2-R. Thus, 
the effects of the support system on the dependent variables for concentration and or- 
der-effects could not be attributed to pre-experimental group differences. For the vari- 
able concentration performance and processed target objects, the interaction effect of 
Support and Working Order reached statistical significance. The group, which first per- 
formed the work task without support, followed by the supported condition performed 
better, compared to the group starting in the supported condition. Based on two of the 
three dependent measures derived from the d2-R, this may indicate that the beneficial 
effect of support on concentration was strongest, when participants had already per- 
formed the work task once before. This could be due to practice-effects regarding the 
experimental setting, or differences between the groups, that could not be ruled out with 
the baseline measurement. As displayed in Figure 3 none of the values of the supported 
conditions were significantly lower than in the unsupported condition of each group (it 
must be noted that the interaction effect of error frequency in the d2R did not reach 
statistical significance). Therefore, comparing the two conditions (supported, unsup- 
ported) within each group, the performance was enhanced for every variable in the sup- 
ported condition. In this context, it must also be discussed whether the above-mentioned 
results of the experiment can be attributed to the fact that participants took significantly 
more time to perform the work task in the supported condition than in the unsupported 
condition. Similar response patterns were commonly found in a variety of tasks, allow- 
ing the person to set a criterion for speed at the expanse of accuracy, termed speed- 
accuracy trade off (for an overview see [22]). To eliminate this possible covariate, the 
differing time spent on tasks with and without the exoskeleton should be aligned in 
subsequent studies, for example with a familiarization phase, as the blocks might have 
been too short to adapt to the unfamiliar exoskeleton. Nevertheless, as the effect sizes 
for the benefit of support were large and the prior study [19] found the beneficial effect 
of support on concentration and errors without speed differences between the condi- 
tions (supported, unsupported), the speed-accuracy trade-off should not account exclu- 
sively for supporting effect of the exoskeleton on concentration enhancement and error 
decrement. 
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5 Conclusion 


The exoskeleton Lucy improved the working results of the participants in the above 
study. A large number of further studies on this topic is necessary to achieve an opti- 
mized fit between user and technical support systems in the future. 

The present study showed that a technical support system in this particular setting 
presents a meaningful addition to the workforce, with regards to a higher concentration 
capacity and error avoidance in comparison to working without such a system. In addi- 
tion, mechanical relief may contribute to health-promoting work. The urgency to inves- 
tigate this topic area further with relevance to social and economic contexts. Psycho- 
logical research may also play a greater role in the development and evaluation of ex- 
oskeletons, emphasizing the importance of interdisciplinary approaches. 

However, errors in the work process can never be completely avoided in practice 
and physically demanding tasks and occupations, such as those in the scaffolding in- 
dustry, will likely remain unchanged in the future. Nevertheless, the results show that 
the use of technical support systems may be considered in the industrial sector as a 
whole, under aspects of concentration levels and error avoidance. 
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Abstract. Within the increasing digitalization, the widespread application of 
modern information and communication technologies and the technological 
ability to systematically and comprehensively capture and store data allow to 
build data storages of unprecedented size and quality. The evaluation and 
efficient use of the implicit knowledge in the data to support decision-making 
processes is becoming increasingly significant in manufacturing companies. 
Thus, new requirements arise for the qualification and competence development 
to efficiently solve engineering applications and issues in manufacturing and 
assembly with advanced data-driven methods. This paper presents the 
contribution of a qualification concept for Machine Learning in industrial 
production that has been realised within a recent research project funded by the 
Federal Ministry of Education and Research. This concept has been designed and 
validated within the university curriculum for graduate students in mechanical 
and industrial engineering, computer science and statistics. Taking into account 
the current challenges in manufacturing and assembly, the contribution of this 
enhanced interdisciplinary competence development can be considered quite 
significant. The results, findings, and future enhancements are presented within 
this paper. 


Keywords: Competence Development, Industrial Data Science, Industrial 
Manufacturing, Interdisciplinary Competence, Machine Learning 


1 Introduction 


The German manufacturing sector employs eight million people, making Germany one 
of the major business and industrial locations worldwide. However, the production en- 
vironment is subject to constant change through ever evolving trends and challenges. 
Alongside high productivity and profitability, a highly diverse and customizable pro- 
duct portfolio has become a central strategic competitive factor for manufacturing com- 
panies [1]. This, however, entails more flexible and increasingly complex manufac- 
turing and assembly systems, pushing traditional methods and techniques progressively 
to their limits. 
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At the same time, the increasing digitalization and the technological ability to sys- 
tematically and comprehensively capture and store data allow to build data storages of 
unprecedented size and quality [2]. This data, however, can be used for optimization 
and decision-making processes within production. By discovering non-trivial and yet 
unknown structures and correlations, Machine Learning (ML), as an interdisciplinary 
subject of computer science and statistics, provides intelligent and automated pro- 
cessing of large amounts of data [3]. While data analysis alone, however, contributes 
to solving practical problems only to a limited extent, it represents, in particular in con- 
nection with the domain-specific expert knowledge, a future success factor of the com- 
panies [4]. Therefore, methodological competence in the field of ML as well as a good 
technical understanding of practical engineering questions are required [5]. This inter- 
section of computer science, statistics and engineering constitutes the field of Industrial 
Data Science (IDS). 

However, companies often lack the resources to adequately assort such interdiscipli- 
nary teams for IDS projects. In order to solve this dilemma, the qualification of aca- 
demic graduates and continuing education of industrial professionals must be modified 
accordingly. [5] Therefore, the research project “InDaS”, funded by the Federal Minis- 
try of Education and Research in the program “Qualification programs and research 
initiatives in the field of Machine Learning”, focuses on the development and validation 
of a comprehensive and interdisciplinary qualification concept of methodical compe- 
tences in the field of ML with practical applications in industrial production. 


2 State of the Art 


2.1 Machine Learning in Production 


Machine Learning (ML) has proven to be very advantageous in various fields of appli- 
cation. Thereby, the success is encouraged by the invention of increasingly sophisti- 
cated ML models [6], the availability of large data sets [7] and the development of 
software platforms [8] which enable an easy employment of vast computational re- 
sources for the training of ML models on large data sets. 

In industrial production, ML methods can be used to overcome existing problems 
and boundaries in a number of application scenarios along the production process chain. 
Successful applications can be found for various tasks in production planning and op- 
timization [9], quality improvement and prediction [10, 11], predictive maintenance 
[12] and energy efficiency optimization [13]. The data used ranges from master data, 
over transaction, log and sensor data to text, voice, video and audio data [14]. Algo- 
rithms applied include supervised and unsupervised methods such as tree-based classi- 
fiers, Support Vector Machines (SVM), Neural Networks, Generalized Linear Models 
(GLM), Clustering techniques, etc. [15]. 

Nevertheless, the existing research gap is a lack of interdisciplinary competence that 
allows the appropriate formulation of learning tasks and application of algorithms for 
engineering questions in a situations-oriented manner. 
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2.2 Educational Concepts 


As mentioned above, an increasing technical and functional competence to handle large 
amounts of data in production in the era of Industry 4.0 is needed [16]. In this context 
and in conjunction with the fact that learning is not directly observable, a multitude of 
divergent theories, models, and approaches emerged to explain human learning [1]. 
Educational concepts that focus on knowledge generation and transfer for individual 
target groups are equally diverse. 

Practically-oriented approaches such as learning factories have become highly pro- 
mising for knowledge and innovation transfer in academic as well as professional edu- 
cation [17]. Teaching and learning is based on actual problem situations and issues from 
research and industry as well as on practice-oriented case studies and defined didactic 
methods [18]. Thereby, the interconnection of research, industry and education enables 
the establishment of interdisciplinary, multi-dimensional teaching and learning 
experiences. As a result, a sustainable transfer of knowledge, the development of com- 
petence and the ability of creative and self-organized action can be captured [18, 19]. 

This kind of approach is also required in order to transfer the essential knowledge 
and skills of IDS to industrial manufacturing and assembly. Through a strong practical 
orientation, the ability to apply adequate methods to real questions is trained in addition 
to the mere imparting of knowledge. There is a comprehensive range of seminars, 
trainings and certificates for various fields of ML including visual analytics, data prep- 
aration and data quality, ML algorithms and methods as well as others [20-25]. Never- 
theless, each of these qualification measures is based on a specific methodological topic 
and does not enable the attendees to experience the extent of actual industrial ML pro- 
jects. Table 1 shows a comparison of different lectures for Data Science and their fo- 
cuses. 


Table 1. Comparison of Data Science seminars concerning content alignment 
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Table 1 shows that, with regard to the seminars offered on the subject of Data Science, 
there is little or no reference to industrial application and the introduction of 
engineering domain knowledge. In addition, the consideration of data management and 
the processing of real use cases is only occasionally part of the seminars. 


3 Concept and Development 


In order to sustainably strengthen the application and dissemination on ML in industrial 
production, competence development and enhancement may not be limited to industrial 
professionals, but address young graduates in university education in particular. 
Thereby, qualification requirements arise from current trends and developments in in- 
dustry as well as the state of knowledge and skills of professional experts. To design 
the education and competence enhancement as efficiently and practice-oriented as 
possible, a mature concept taking all given requirements fully into account is a neces- 
sary precondition. 


3.1 Qualification Requirements for Industrial Data Science 


Future concepts for qualification and competence development are exposed to in- 
creased requirements and must be adapted and supplemented by additional aspects in 
order to guarantee sufficient competencies in the age of digitalization. The rising com- 
plexity of industrial processes requires an increase in specialist competencies, while the 
greatest challenge is to transfer theoretical knowledge to practical questions and areas 
of application [5]. Therefore, new approaches should include practical applications and 
experimentation so that a comprehensive understanding can be created. 

A survey by Bauer et al. in 2018 revealed, that only 5 out of 57 companies (9%) 
within different industrial sectors such as automotive, electronics and mechanical en- 
gineering, already use ML widely [5]. Further 27 companies (47%) stated that they had 
gained initial experience or used methods ona small scale. The remaining 25 companies 
(44%) have not yet used ML, but plan to do so in the future. 

Thereby, the lack of corresponding methodological competence has been identified 
as the major obstacle for the use of ML. To overcome this shortcoming, a catalogue of 
requirements must be fulfilled by adequate qualification and competence development 
concepts for Industrial Data Science [5]. Based on didactic-methodological knowledge 
transfer, the subjects of data management, ML and domain knowledge are the threefold 
foundation of IDS education. These core topics can be illustrated by three overlapping 
thematic areas (see Fig. 1). 
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Fig. 1. Concept for qualification and competence development of Industrial Data Science as 
part of the research project “InDaS” [5] 


3.2 Conception of interdisciplinary competence development 


Within the research project "InDaS", the aforementioned requirements are addressed 
in a strongly practice-oriented and interdisciplinary qualification concept for academic 
graduates and industrial professionals. This concept combines engineering knowledge 
and skills with methodological expertise in statistics and computer science. Through 
close cooperation with manufacturing companies and an early integration and 
consideration of practical requirements and applications, a high practical relevance of 
taught contents is sought. The processing of industrial use cases by students with 
supervision by industrial professionals and university lecturers represents a new 
approach to teaching. Students work on problems in a real context and realistic levels 
of difficulty and make contact with companies during their studies. The educational 
offer addresses students aiming for a master’s degree in mechanical and industrial 
engineering, computer science or statistics as well as industrial specialists. The course 
is divided into a theoretical phase to first convey basic contents and a practical phase 
to apply gained knowledge to practical use cases and questions. 


3.3 Development of Methodological Competence 


The basic theoretical contents have to be imparted to enable an independent processing 
of industrial questions with appropriate methodologies. Hereby, the selected theoretical 
contents are determined by the specific needs of industry to react to current trends and 
changes. With regard to the processing of large data sets, many companies criticize the 
manual acquisition and processing of unstructured data and the data administration 
[26]. Thereby, especially large data volumes are challenging and a faster data supply is 
desired. Further identifiable problems include a lack of data cleansing and analysis ca- 
pabilities, a non-agile business intelligence infrastructure, a lack of access to data 
sources, a poorly intuitive user interface and errors in data management. [27] 

To address the before mentioned issues and requirements, the curriculum comprises 
the thematic modules of data management, machine learning and engineering domain 
knowledge, built on the basis of didactic-methodical knowledge transfer (see Fig. 2). 
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Fig. 2. Contentual and didactical design of the IDS qualification concept 


In the theoretical part, complex contents are reduced to their general idea in order to 
create a basic understanding of multiple methods and approaches. In addition, potential 
fields of application, problems and limits of learned methods are discussed to enable 
the students to take them into account in the later practical implementation. Within the 
experience-based learning approach, learning takes place in interdisciplinary working 
groups with the inclusion of industrial application cases as well as a project-oriented 
development of individual solutions. 

The efficient management of data requires knowledge in the areas of database sys- 
tems, data administration, resource management and cleansing and processing of struc- 
tured and unstructured data. To enable the students to practically realize these tasks, 
experiences with cloud solutions, database concepts, and script languages, e. g. Python, 
R or RapidMiner, for modelling and programming data science solutions are taught. 

In addition to mere knowledge of different ML methods and approaches, their de- 
mand-oriented application is an essential success factor and is therefore explicitly taken 
into account in IDS qualification. Contentually, the processing pipeline of data prepro- 
cessing, modelling, validation and evaluation is covered through the focal points of data 
visualization, feature engineering, ML algorithms, validation and evaluation. 

From the engineering point of view, it is essential to understand the manufacturing 
and assembly processes to detect the potential of ML for individual business units. To 
execute ML projects systematically, respective process models such as CRISP-DM 
(cross-industry standard process model for data mining) [28] should be known. Addi- 
tionally, domain-specific structuring models, e. g. the product life cycle, are needed to 
examine the achieved results for their relevance and cost-effectiveness regarding the 
underlying business case. 

Based on the mentioned modules, the students are capable of solving industrial tasks 
with data-driven methods and developing innovative solution and improvement ap- 
proaches in manufacturing and assembly. They are able to lead and support interdisci- 
plinary project teams and to pass on their interdisciplinary competences and skills. 


3.4 Transfer to Practical Applications 


It is an essential success factor to ensure that young academic graduates and industrial 
professionals not only know the theoretical backgrounds of IDS, but also build up a 
deep understanding and are able to apply methods situatively and correctly. Therefore, 
the second part of the qualification concept consists of the practical processing of in- 
dustrial case studies in interdisciplinary working groups. Under the involvement of in- 
dustrial specialists, different case studies covering ML applications such as pattern 
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recognition and image classification (see Fig. 3) enable the enrichment of theoretical 
knowledge with practical experience through knowledge transfer and experience-based 
learning. 
Provided 
data formats 


= Automatic quality evaluation of Classification of defective 
= injection molded parts injection molded parts 


e Classification and control of Evaluation, forecasting and 
chemical batch processes optimization of process behaving 
EN) @:% Identification of field failures Identification of potentialreturns 
based on EOL measurements and responsible parameters 


FERN Reduction of pseudo faults in the Automatic classification of 
aT se control of plug connection incorrectly assembled connectors 


Fig. 3. Case studies of the 2019 practical IDS seminar at TU Dortmund University 


Objective Methods applied 


After the general introduction of the use cases with support of industrial process 
experts, the project takes place in form of a three-month self-organized, independent 
group work with regular scientific supervision by industrial and methodological ex- 
perts. The groups compose of graduate students from the disciplines of statistics, com- 
puter science as well as mechanical and industrial engineering, ensuring interdiscipli- 
nary key competencies. The results are presented in a joint seminar, including a discus- 
sion of results and outlook to further research activities. 


4 Concept Evaluation 


The evaluation of the concept is based on the mandatory course evaluation as well as 
the results of the learning progress assessment. 

A 60-minute written examination at the end of the theoretical phase serves as a 
learning success control. The imparted topics of data administration, statistical basics 
of ML, ML methods and engineering domain knowledge are queried in one assignment 
each, consisting of several subtasks. The degree of complexity ranges from simple 
inquiry of knowledge, the transfer onto new facts and conditions and the linkage of 
information to solve more complex problems. The results of the examination and the 
results of the anonymous evaluation in the form of a written questionnaire point to 
identical findings. 

The results and feedback from students in all three subject areas show that the com- 
prehensibility of the contents from the individual disciplines has been of a similar high 
standard. As a result, it can be concluded that the preparation of the content can be 
regarded as quite successful, taking into account the heterogeneous target group [29]. 
The only criticism received has been directed to the introduction to the programming 
language R, which has been perceived as too condensed by students without prior 
knowledge. Great interest in the IDS education has been reflected among the university 
students as well as on the side of industrial partners. As a key figure, the number of 
participants which increased from 51 in the winter term 2018/19 to 71 in the winter 
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term 2019/20 (+39%) can be mentioned. A survey among the cooperating industry part- 
ners has revealed that all of them rate the achieved results as above their expectations 
and are willing to cooperate again in the next practical seminar. 


5 Conclusion and Outlook 


Enabled by the increasing digitalization, the widespread application of modern infor- 
mation and communication technologies and the availability of large amounts of data, 
ML applications are becoming increasingly significant in industrial manufacturing and 
assembly. To address the associated qualification requirements, new educational con- 
cepts have to be developed. The “Industrial Data Science” qualification concept enables 
an interdisciplinary competence enhancement on the intersection of computer science, 
statistics and the engineering domain. Through the combination of teaching of theore- 
tical contents and the transfer to practical applications, a comprehensive and practi- 
cally-oriented qualification is facilitated. The learning success and feedback from the 
concept evaluation in university education at TU Dortmund University has shown that 
the course addresses current industrial questions and requirements and is subject to high 
interest. With the goal of integrating potential benefits into manufacturing companies 
sustainably, the qualification concept has to be expanded and continuously improved 
in order to guarantee ongoing actuality. Therefore, the qualification in IDS must be 
anchored in the university curriculum for graduate students of relevant disciplines such 
as statistics, computer science and especially engineering and the target group has to be 
expanded onto skilled professionals in manufacturing companies. 
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Abstract. In the production of offshore tubular connections, the cutting process 
is an important factor. Undefined root gaps between the individual pipe connec- 
tions require time-consuming reworking and increased welding effort, resulting 
in longer production times and higher costs. The developed procedure is based 
on the 3D measurement of the pipes in order to consider the real geometries in 
the planning of the cutting process and to optimize the following joining process 
by adapting the root gap all over the tubular connection and to get a constant 
seam cross section. The basic concept is to build the planned connection virtually 
with the measured and real pipe geometries and to plan the flame cutting process 
including weld seam preparation on the base of the result. For the measurement 
in the production environment, a suitable measuring concept was developed to 
return the results directly into the cutting process with the robot. This is the base 
for the automation of the tubular connection production. 


Keywords: offshore structures, sensor-based programming, path planning, ro- 
botic cutting 


1 Introduction 


Jacket structures play an important role in offshore technology. They are used as foun- 
dation for converter platforms at sea as well as for wind energy plants. Jackets are com- 
plex tubular connections based on pipes (Fig. 1). Compared to other offshore founda- 
tion structures, the used pipes have smaller diameters and material thicknesses. Accord- 
ingly, jackets can be produced with lower material costs than comparable structures 
with similar functions, such as monopiles. However, the complexity results in higher 
production costs. An essential cost driver are the deviations in the pipe production, re- 
sulting with time-consuming and cost-intensive manual rework in the assembly. 
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Fig. 1. Jacket structure (left), tubular connection (right) [1] 


The main reason for the need for reworking was identified as the determination of the 
cutting contours without taking into account the production deviations of the real pipes. 
By measuring the real tube geometries and by not using the CAD geometry, the manu- 
facturing process for joining the tubular connections is to be adapted in order to mini- 
mize manual reworking and thus reduce manufacturing costs. 

Fig. 2 shows the manufacturing process of a tubular connection as a process chain. The 
process adapts the cutting process to the tolerances in pipe production and serves as the 
basis for positioning and joining the pipe. 


| joining/ 
pipe cutting E positioning > j 


production process welding 


Fig. 2. Process chain for tubular connection production 


Tubular connections consist of a main member pipe and one or more branch member 
pipes (see Fig. 3), whereby the pipe diameters used here varied from 500 mm to 
3000 mm and the material thicknesses between 20 mm and 120 mm. 

By flame cutting, the pipe of the branch member is prepared for the joining process. 
The contour which is calculated by virtually placing the branch member pipe on the 
main member pipe is based on the ideal geometries of both pipes. In addition, the weld 
seam preparation is planned for the branch member pipe by using this contour. For 
conforming the standards of joining for this tubular connection it requires a defined root 
gap between 2 and 8 mm [2], which depends on the different zones over the various 
areas of the weld (see I, II and III in Fig. 3). 
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Fig. 3. K-connection with definitions of root gap, dihedral angle (opening angle between sur- 
faces) and the zones (I — heel zone; II — side zone; III — toe zone) 


main member 


Due to the production deviation of both pipes, the ideal seam course in the pipe joint 
must be adapted to reality. If this is not the case, the joining technology will not be able 
to tolerate root gaps, which will lead to serious manual reworking such as grinding of 
the weld in the cut surface. The conventional manufacturing of tubular connections 
doesn’t consider the real geometries of the main member pipe during weld seam prep- 
aration. Only the real pipe geometry of the branch member is used to ensure the quality 
of the cut. 

By planning the contour based on real pipe geometries and by generating constant 
seam cross sections in the weld seam preparation, automated welding with the specially 
developed orbital welding system in high quality is possible [3]. 


2 Description of the developed process 


In order to achieve these goals, the manufacturing process has to be adapted and the 
measurement of the pipes integrated into it. Specifically, the following steps (Fig. 4) 
are necessary: 


Recording the geometry of the main and the branch member 
Determination of the virtual cutting contour 

Inclusion of weld seam preparation 

Flame cutting of the pipe with a robot 


Compared to the state of the art, the first two steps reduce the expensive and timecon- 
suming re-working by a slightly increasing production planning. 
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Fig. 4. Process steps for cutting with robots 


In the following, the mentioned four steps are examined using an example tubular 
connection. In the selected configuration, a branch member pipe with a diameter of 
508 mm and a material thickness of 20 mm is placed centrally at an angle of 45° on the 
main member pipe with a diameter of 1016 mm. 


2.1 Test setup 


For the execution of tests, a suitable plant design was developed. A welding robot 
(KUKA kr5 arc) in combination with a turn and tilt positioner (KUKA DKP400) is 
used to extend the working area of the robot. The positioner can handle pipes up to a 
mass of 400 kg, which is necessary for the chosen branch member pipe. 


reference target 


flame torch 


IgG KUKA kr5 arc 


KUKA DKP400 /- 


Fig. 5. Test setup for the robot cutting 


177 


In order to work with the highest accuracy during the entire process, the coordinate 
systems of the tilt and turn positioner and the robot were measured to each other using 
an external measuring system (laser tracker). After the robot system was measured, it 
was referenced to a selected global system defined by reference marks. This is used 
throughout the process. 


2.2 Digitization of the pipes 


Both pipes are digitized in the production environment by a laser scanner. The used 
Z+F IMAGER® 5010 has a linearity error of less than 1 mm and a distance noise of 
0.5 mm rms at 10 meter [4]. Distances of more than 10 meter to the measuring object 
are not provided for in the measuring concept. 

The main member pipe is measured horizontally. The coordinate system is defined by 
the pipe axis, flange plane and a marking of the theoretical intersection point of the pipe 
axis with the pipe surface. 

When measuring the branch member pipe, it is perpendicular to the tilt and turn posi- 
tioner. The robot has a pose repeatability of + 0.06 mm [5], which is used throughout 
the process. Instead of measuring the tube from different angles and positions, only one 
laser scanner position is required and the pipe is rotated between each scan using the 
tilt and turn positioner (see Fig. 6). The registration of the scans takes place in the co- 
ordinate system of the tilt and turn positioner, defined by the rotary axis of the table 
and a rotary angle. The generated point clouds of the outer surface of the branch mem- 
ber pipe are assembled in the evaluation software by turning them back about the de- 
fined rotary angle of the positioner. Due to the overlapping areas of the various scans 
and the unfavorable impact angle of the laser, data smoothing is necessary. 


s; suun 
232227 npp 


Fig. 6. Measuring setup for the measurement of the branch member pipe 
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A measurement of the branch member pipe from the inside does not take place, since 
the effort seems inappropriate due to the inner diameter and difficult referencing. Dur- 
ing preparatory measurements of the pipe geometries, it was determined that the mate- 
rial thicknesses of the pipes, with the exception of the weld seam, is approximately 
constant. Therefore, the data points of the inner pipe surface are simulated on the base 
of the data points of the outer pipe surface using the pipe axis determined by cylinder 
segmentation according to [6]. The planned configuration is now virtually simulated 
with the point clouds of the main member pipe and the branch member. The axis of 
branch member pipe is transformed to the z-axis and the axis of the main member pipe 
is aligned at the defined angle (see Fig. 7 (left)). The breakthrough point of the z-axis 
on the surface of the main member pipe is known from the measurement concept. By 
translating the point cloud parallel to the pipe axis, the dimensions of the future socket 
are virtually taken into account. 


Fig. 7. Process flow for determining the virtual intersection contour using the example of the 
outer surface of the extension tube, virtual alignment (left), polygonised intersection areas (cen- 
ter) and intersection contour (right) 


Using trigonometric methods, the intersection areas of the point clouds were identi- 
fied on the base of the configuration and the points lying within them were polygonised 
(see Fig. 7 (center)). From the calculation of the section of the polygonised areas, a 3D 
polyline is obtained, with a C? continuous, closed BSpline curve approximated by its 
straight line center points (see Fig. 7 (right)). The results are the curves (inner and outer 
contour), in which point cloud of the main member pipe crosses the point cloud of the 
branch member pipe. The outer curve corresponds to the seam contour (see Fig. 3), on 
which the weld seam preparation takes place. 


2.3 Weld seam preparation 


The dihedral angle is used to determine the seam course [2]. Fig. 8 shows a seam course 
corresponding to the standard for the production of a pipe joint with the used configu- 
ration. If the dihedral angle is smaller than 50°, a fillet weld is prepared (zone I, Fig. 3), 
if it is larger than 90°, a bevel groove is prepared (zone II, Fig. 3). The area in between 
is called the transition area (zone II, Fig. 3), because the change between these two 
seam types takes place. The course of this seam was defined in such a way that it can 
be flame cut with a single move. The challenge is that the seam opening angles which 
needs to be cut becomes acute. For later use, this step has to be adapted to a two-part 
cut. 
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Fig. 8. Seam definition in the selected configuration, depending on the dihedral angle 


The shown dihedral angles were determined with the seam contour of an ideal tub- 
ular connection with faultlessly pipes. In case of a fillet weld, the main member pipe 
and the branch member pipe touch each other at the outer contour. In case of a bevel 
groove, the inner contour of the branch member pipe is in contact. 


transition area fillet weld 


bevel groove 


Fig. 9. Target layer structure for tubular connection at different areas (branch member — top, 
main member — bottom) 


The necessity of an accurately planned cut and the contact between both pipes is 
shown in Fig. 9. For a future automated welding process, with the aim of achieving a 
uniform layer structure, it is essential to achieve a constant root gap and a defined weld 
cross section. Otherwise, the welding parameter has to be adjusted during the entire 


joining process. 


2.4 Offline programming and Preparation for flame cutting 


Since the flame cut is placed on the outer surface of the branch member pipe, the cal- 
culated points for the robot have to be shifted to the outer surface of the branch member 
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pipe along the cutting direction (see Fig. 10 (left)). For a constant cutting quality, the 
torch distance to the cutting surface has to be constant. This is achieved by further 
shifting each point along the corresponding cutting direction by an offset. A retransfor- 
mation of the data provides the discrete positions and directions of the torch head in the 
coordinate system of the tilt and turn positioner. The robot is integrated in this coordi- 
nate system. Since the tilt-turn positioner has a higher accuracy than the robot, the turn- 
table rotates during the cutting process, while the robot only performs small move- 
ments. The robot software interpolates between the position and direction data and cal- 
culates the positions of the axis in order to avoid singularity positions. By using syn- 
chronized movements from the positioner and the robot, it is possible to cut at constant 
speed using all eight axes. 


calculated cutting 
contour (offset) 


outer contour 


inner contour 


Fig. 10. Calculated contour (left) and path planning with the robot in SIEMENS RobotExpert 
(right) 


This is the base for the offline-programming of robot motion (see Fig. 10, right) and 
collision control to prevent crashes between the robot/torch and the pipe. In addition, 
motion commands and switching commands for the cutting process, such as start and 
stop distances, ignition commands and waiting times, have to be added. 


3 Verification and Results 


The method is used to manufacture the described tubular connection. The cut branch 
member pipe, especially the cut surface, is measured with the strip light projection sys- 
tem GOM Atos Triple Scan. A 300 x 300 mm? measuring field is used, with which a 
measuring accuracy of approx. 0.01 mm can be achieved [7]. 

First of all the resulting model is compared with an ideal cylinder to determine the 
production deviations. On the other hand, the model is virtually placed on the digitized 
main member pipe in the defined configuration and aligned by best-fit method, whereby 
only rotation around and displacement along the pipe axis are permitted. The root gap 
can be derived from this. The results are shown in the form of a measurement in Fig. 11. 


Fig. 11. Measurement results to illustrate the production deviations of an extension tube (left) 
and to illustrate the deviation of the cut surface from the planned one (right). 


The production deviations of the attachment pipe are clearly visible in the left meas- 
urement. The tolerance of +5 mm is maintained [8]. The right measurement shows the 
deviations between the virtually planned and the actual cut surface. The maximum de- 
viation, this means the largest gap between both pipes, is 1.59 mm. The minimum de- 
viation, the largest penetration of the two pipes, is -1.83 mm. They vary by 3.42 mm. 
Within the bevel groove area (zone III, Fig. 3, upper cut area, Fig. 11) the deviations 
differ by less than 1 mm. Within the fillet weld area (zone I, Fig. 3, lower cut area, 
Fig. 11), too. The variations in the transition areas are more than 2 mm. The specified 
fluctuations of the nominal root gap of 6 mm are adhered to. Despite the production 
deviations, the weld seam can be prepared in accordance with the standards. 


4 Conclusion and outlook 


It has been proven that the interaction of measurement technology, cutting contour 
planning and robot cutting can be successfully applied in practice. Both the tubular 
connection configuration and the selected seam configuration are to be understood as 
an example. The process can be used for any tubular connection and seam configura- 
tion. Further optimization possibilities are in the next process steps of positioning and 
joining (see Fig. 2). The challenge in positioning is that the approach point planned in 
the configuration, in which the branch member pipe axis meets the axis of the main 
member pipe, is calculated and has to be transferred to the real pipe configuration. 

In order to optimize the joining process, based on a two-part cut, a suitable seam con- 
figuration can be developed. Because the seam course is planned and implemented on 
the base of the actual data, virtual weld layer planning is possible without further meas- 
urement, which additionally accelerates the production process. In particular, this ena- 
bles automation of the welding process. 


182 


References 


1. Brauser, S, Lauer, S.: Mechanisierung eines Mehrlagenschweißprozesses zum Fügen von 
Offshore-Rohrverbindungen, 8. Zukunftskonferenz: Wind & Maritim, Rostock 2019 

2. American Welding Society: AWS D1.1:Structural Welding Code-Steel, Miami (2000) 

3. Herholz, H., Dryba, S., Griindler, M., Wanner, M.-C.: Development of a flexible special 
kinematics for fully mechanized production of tubular nodes for offshore foundation struc- 
tures. In: Tagungsband des 2. Kongresses Montage Handhabung Industrieroboter, pp. 1-11. 
Springer, Heidelberg (2017) 

4. Zoller + Fröhlich GmbH, Z+F IMAGER 5010 Datenblatt, Bridgewille:Z+F USA 

5. KUKA Roboter GmbH, KUKA Positionierer DKP-400, DKP-400 endlos Spezifikation, 
Augsburg 2013 

6. Rusu, R. B., Cousins,S.: 3D is here: Point Cloud Library (PCL), Menlo Park 2011 

7. GOM GmbH, Atos Triple Scan — Industrieller optischer 3D-Digitalisierer, Braunschweig 
2017 

8. Deutsches Institut für Normung: DIN EN 10216-1:2014-03 Nahtlose Stahlrohre für Druck- 
beanspruchungen — Technische Lieferbedingungen — Teil 1:Rohre aus unlegierten Stählen 
mit festgelegten Eigenschaften bei Raumtemperatur, Beuth Verlag 2014 

9. Ambrosat,T., Lauer,S., Geist,M. Flügge,W.:Bestimmung komplexer Schneidkonturen als 
Vorbereitung zum Verschweißen von 3D Rohrstößen, GFal, Berlin 2018 


Open Access This chapter is licensed under the terms of the Creative Commons Attri- 
bution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which 
permits use, sharing, adaptation, distribution and reproduction in any medium or for- 
mat, as long as you give appropriate credit to the original author(s) and the source, 
provide a link to the Creative Commons license and indicate if changes were made. 

The images or other third party material in this chapter are included in the chap- 
ter’s Creative Commons license, unless indicated otherwise in a credit line to the 
material. If material is not included in the chapter’s Creative Commons license and 
your intended use is not permitted by statutory regulation or exceeds the permitted 
use, you will need to obtain permission directly from the copyright holder. 


Information requirements and interfaces for the 
programming of robots in flexible manufacturing 


Arturo Bastidas-Cruz*, Oliver Heimann*, Kevin Haninger, and Jörg Krüger 


Fraunhofer Institute for Production Systems and Design Technology (IPK) 
Division of Automation Technology, Berlin, Germany 
firstname.lastname@ipk.fraunhofer.de 
* Equal contribution 


Abstract. Flexible automation limits what information is known during 
robot commissioning, requiring new robot programming methodologies. 
To adapt robot behavior to product variation, an operator can supply 
missing information, controlling predefined robot behaviors via a user in- 
terface. This operator interface can abstract domain expertise (in robotic 
programming, assembly planning), allowing the efficient specification of 
changes to a robot program by a wide range of potential users. Towards 
designing such interfaces, this paper analyzes the requirements of flexi- 
ble manufacturing, the required changes to the robot program, and the 
information needed to make those changes. Two user interfaces are pre- 
sented, a drag and drop and a gesture interface, implemented on a robotic 
flexible manufacturing testbed. 


Keywords: flexible manufacturing, robot programming, human-robot 
interaction, usability 


1 Introduction 


In standard industrial robot commissioning, as common in large-scale assembly 
lines, experts produce a fully-defined sequence of robot commands for a specific 
task during commissioning. Although providing higher quality and throughput, 
traditional automation reduces production agility with limited ability to accom- 
modate product variation or new products without recommissioning. For small- 
and medium-sized companies in saturated markets with a faster product cycle, 
this is a major barrier. In comparison, on manual assembly lines new prod- 
ucts can be quickly deployed (provided no hardware changes are required) and 
product variation is tolerated with marginal increases in cost or deployment 
time. Though difficult for standard automation, flexible manufacturing can ad- 
dress many growing business needs: individualized products, smaller lot sizes, 
shorter innovation cycles, and shorter production times. These capabilities sup- 
port onshoring; local production at competitive costs. Flexible manufacturing 
can involve product variation, where the product varies between task iterations, 
or production changeover, where the assembly line is reconfigured for a new 
product. The manufacture of multiple products on an automated line can be 


© The Editor(s) (if applicable) and The Author(s) 2020 
T. Schiippstuhl et al. (Hrsg.), Annals of Scientific Society for Assembly, 
Handling and Industrial Robotics, https://doi.org/10.1007/978-3-662-61755-7_17 


® 


Check for 
updates 


184 


solved centrally, through methods like a production management system, which 
indexes all products and adjusts a cell as necessary. Decentralized flexibility is 
also possible, through the design of adaptive cells which detect and respond to 
product variation (e.g. with vision systems, or identifying tags). However, these 
approaches are complex and do not reduce the effort to introduce a new product. 
A promising approach to flexibility is human robot collaboration (HRC), where 
human operators can respond to product changes, adapting pre-programmed 
robot behaviors as appropriate [7]. Designing the robot such that an operator 
can later complete a task requires consideration of what decisions the human 
must make, how those decisions are communicated to the robot, and the con- 
textual information required to make those decisions. Programming for flexible 
automation, moreso than traditional automation, occurs over time, where a va- 
riety of users contribute to the final sequence of robot commands. From the user 
perspective, the interface must include visibility of system status, understandable 
information presentation, a match between the system and real world, and help 
users recognize and recover from errors [2], [6]. Several modes of human-robot 
communication for modifying a robot program have been developed, including 
speech [9] and gestures [10], [11], [13]. However, more intuitive and compact 
modalities of communication can have ambiguity in the interpretation of the in- 
put. Therefore, the robot and human must have a common understanding of the 
environment to correctly ground an exchanged symbol (see [14] or [12]). This is 
a usability requirement for the design of intuitive human machine interfaces. 


Human-robot communication can also be achieved through a graphical inter- 
face on a touchscreen in the workspace. Such graphical environments can allow 
for the simultaneous presentation of more complex information, allowing a user 
to see and adjust the parameters and program flow of a robot. This interaction 
can be further improved through the use of augmented reality (AR). AR al- 
lows GUIs to present information as an overlay to the actual evironment, which 
enables a strong visual connection between parameters and their real world in- 
fluence. Lambrecht [10] et al have shown that tablet based augmented reality 
can significanlty reduce the programming time for free space programs. Guhl [4] 
et al as well as Ong [5] et al have further improved on the approach through 
a headworn AR device allowing for comprehensive virtual interaction with the 
robot program. 


This paper explores HRC in flexible manufacturing, examining application 
requirements and usability perspectives. Programming for flexible manufactur- 
ing is presented as the partitioning of expertise and process information within 
a prior robot program, where the information required is reduced and easily 
supplied. While other GUI development considers defining a robot task from 
lower-level commands [3], the GUIs discussed here are targeting flexible manu- 
facturing, where a subset of decisions or changes must be made via the GUI. Two 
interfaces for the flexible programming of a robot cell are presented, a gesture- 
based system for workspace communication, and a touchscreen-based graphical 
interface with predefined actions. 
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2 Flexible Manufacturing 


What new design requirements does flexible manufacturing introduce? Consider 
a tier two supplier for the car manufacturing industry, where different subassem- 
blies must be produced for just-in-time orders of tier one suppliers. The ordered 
variants will vary between the different customers and on a day-to-day basis, 
requiring fast change over times. Furthermore, some variants might only ever be 
produced a hundred times, rendering the programming of a robotic cell for all 
possible production unviable. Flexible manufacturing can also encompass artisan 
production, where highly individualized products are manufactured and robot 
assistants can be directed in response to material variation, artistic expression, or 
customer individualization. Table 1 presents different aspects of manufacturing 
which may vary in flexible production with examples. 


Table 1: Flexible manufacturing examples, which may require on-/offline changes 
to the robot program 


Variation || Examples Offline|Online 


Product Geometry|\Screw locations 


Workspace Geometry|\Material pickup location 


Inclusion/Omission of Operation||Optional subassembly 


“Is Im | 


Change in Operation Ordering||Production process change 


Artisan Process||Glue application location 


There are two major types of variation considered here: geometric and plan. 
In geometric variation, the location(s) which determine a specific action may 
vary between cell cycles or product change-over. In plan variation, a change in 
the set or order of robot behaviors must be made. Other changes to the robot 
program may be necessary (dwell time, contact force, etc), but are typically 
process parameters which will require more expertise and are outside the scope 
of this paper. 


3 Decision making 


Product variation, as introduced in the last section, requires corresponding 
changes to the robot program. This section overviews available and necessary 
information to make those changes at the various stages of the flexible program- 
ming. 


3.1 Information 


Information and expertise is required to translate a product design into an ap- 
propriate set of robot instructions. As seen in Table 1 the product model changes 
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least frequently; it is a result of the design process and therefore is considered 
available as input to the assembly level planning. The assembly planning requires 
a very high level of expertise to translate a product design into a sequence of 
manufacturing processes in order to assemble the product. It involves knowl- 
edge about machine availability and resource planning. The assembly planning 
problem is well supported by tools (computer-aided manufacturing), more easily 
dealing with high-mix, low-volume production compared to lower-level planning 
layers (e.g. robot programming). 


On the basis of a product-specific assembly plan, a corresponding robot task 
order - the sequence of robot actions - must be produced. Where the assembly 
plan includes general operation orders, the robot task order must specify the 
product-specific robot actions which achieve this result. With increasing variants 
and smaller production lots, this results in an exponential growth in required 
decisions, and tighter time constraints on every individual decision. 


Therefore, a programming system for flexible manufacturing on the task level 
requires the ability to rearrange the order of processes or even substitute different 
processes in an efficient manner. An example is the production of gear boxes: 
while the overall process of interlocking gears, fitting shafts, bearings and seals 
is comparable between products, materials, sizes or even the number of overall 
parts might vary. Here, the operator needs to adapt the overall flow of control 
of a prior program. Furthermore, at the task level, the operator might specify 
specific process parameters. These include feed velocity, maximum torque for a 
screw or the slope of a thread to be cut. In order to make these decisions the 
operator needs expert knowledge regarding the machining processes involved 
as well as the capability to translate product specifications into the required 
processing parameters. 


Decision speed becomes even more important on the individual action level as 
again, per task instance, multiple actions must be specified. Here craftmanship 
is required to determine the correct instantiation of an action on a per product 
basis. As there may be no purpose made fixtures or computer vision systems for 
the individual production lot, the human operator needs to supply this informa- 
tion. Where the operation order is programmed in advance, the artisan process 
requires information in situ. The frequency of online decisions requires efficient 
in-situ programming to avoid a bottle neck in the manufacturing processes. 


Figure 1 gives an overview on the different levels of programming and the 
corresponding informational requirements. It can be seen that the frequency of a 
decision increases as it comes closer to program execution. At the same time, the 
breadth of information required becomes narrower. This increasing specialization 
of the information provided helps to address the need for fast decisions. 


While at the task planning level, an interface needs to provide means to 
program the entire process. This results in the necessity for the user to navigate 
a more complex interface. However, on the process level, the interface might just 
offer a very limited set of options specifically tailored to the individual process, 
allowing very fast navigation and user interaction. 
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Fig.1: The various levels on which a flexible manufacturing task can be pro- 
grammed, where a prior program can be specialized, with discrete or continuous 
inputs. The last two stages (task order and action list) are static in standard 
automation, but here can be adjusted offline or online. 


3.2 Interface Usability 


The prior section has shown that the frequency of required decision making 
varies with the level of detail of the process specification. For frequent shopfloor 
decisions, a user can make adjustments to the robot execution via an HRC inter- 
face. The means by which a user provides this information to the robotic system 
affects the ability to display contextual information, ergonomics, learnability, 
and flexibility [1]. HRC is very much defined by its context, requiring a common 
grounding to the symbols exchanged between robot and human. 

The interface must provide feasibility, the ability to adjust the robot behavior 
as necessary for the appropriate task, and safety, such that for any supplied input 
of the operator the robot maintains safety. On top of the functional requirements, 
the interface should support usability [2], [6] in terms of: 


Visibility: Information required to make a decision is readily available, in- 
cluding relevant states of the robot. 

Predictability: The effects of an input are intuitive and repeatable, so that 
the user can choose the input required to achieve the desired robot behavior. 
Efficiency: The necessary action can be quickly and ergonomically given 
to the robot. For online decisions, this should not significantly disrupt the 
operator’s workflow. 
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4 Implementation 


Fig. 2: Integrated flexible manufacturing demonstrator 


To investigate feasibility and usability of various types of HRC for flexible 
manufacturing, the testbed seen in Figure 2 is built. The cobot used is a Universal 
Robot UR3. Above the table 2 RGB cameras and a projector are mounted. The 
current use case is the automatic placement of workpieces from a storage position 
onto the table followed by a simulated welding operation. This testbed can test 
variation in both geometric and plan information, where decisions must be made 
by an operator online or offline. Two interfaces are presented, which differ in how 
they present the available decisions and contextual information to the user, and 
how the user supplies their decision to the robot. 


4.1 Graphical Programming 


The graphical task-oriented robot programming system presented in this work 
was developed by researchers from the Process Automation and Robotics de- 
partment of the Fraunhofer IPK institute in order to allow visitors without any 
programming experience to interact with the cobot, so that the user can make 
modifications to the robot program and execute an assembly task. The system 
is based on the task-oriented RP-i3 framework developed at Fraunhofer IPK [8] 
and aims to reach easy accessibility of complex systems by workers in industry. 
The task-level framework offers a visual structured interface of the assembly 
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process that allows an easy implementation of the two major types of variation 
discussed in Section 2: an easy rearrangement of the order of processes related to 
a specific assembly task (plan) as well as the easy specification or modification 
of specific action parameters (including geometric variations). 

The user selects tasks that the robot can perform on the work pieces, where 
tasks and work pieces are offered as ready-to-use blocks. The blocks are combined 
to a program by dragging and attaching them to each other. New robot programs 
and new work pieces can be defined in that way. The graphical programming 
environment SCRATCH [17] visualizes the assembly environment abstraction of 
the RP-I3 framework as seen in Figure 3b. 

A robot programming language based on C++ implements the RP-I3 ab- 
straction of tasks and workpieces. A server that works as an interface receives 
the content of the GUI, organize the information in a data base and execute the 
programs using this information. In order to send the command to the robot, 
an external interface is used. The setup consist of a Cobot equipped with a 
two-finger gripper. The RP-i3 framework is located on a Linux-based PC. The 
visualization of the RP-i3 framework is based on a version of Scratch that was 
modified in order to allow the implementation of robot-based assembly tasks. 

This interface offers a higher information-density, allowing the presentation 
of a complete robot plan which helps a user predict robot behavior. Although the 
actions and workpieces are abstracted from the physical workspace, graphics seen 
in Figure 3b can improve the operator’s understanding. However, this interface 
does not as easily allow online monitoring or decisions, as the screen must be 
mounted to not obstruct the workspace. 


Visual support of ibraries for er — 
the subjects and collectionof Area to visualize 


robot actions subjects, robot and build a 
from the actions and robot assembly 
application assembly tasks program 
(a) Graphical Programming (b) User Interface 


Fig. 3: Programming of an assembly task using the graphical interface RP-i3 and 
universal robot UR3 
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4.2 Gesture based Programming using Augmented Reality 


The programming interface for the action level aims to provide very fast and 
precise user input for a very limited breadth of decisions that allows in detail 
specification of the action at hand. Where the RP-i3 GUI enables the speci- 
fication of the more abstract task and its parameters (e.g. weld together part 
A and B with feed velocity of 20mm/s), the gesture based interface allows to 
specify artisan-level decisions (e.g. exact position and length of a seam). This 
information is complex (therefore time consuming and expensive) to describe in 
advance at the task level but very easy to specify once the product is available 
in the manufacturing cell. 

The interface allows machine operators to communicate these decisions in 
a very fast way through direct interaction with the program on the workpiece 
using augmented reality [15]. Geometric information (planar locations) can be 
communicated, as well as discrete decisions via virtual buttons in the workspace, 
allowing the omission of other inputs device such as a touch panel. Furthermore, 
the direct projection of paths onto the workpiece provides the user with under- 
standing of the robot state and plan. 


=y 


Finger ins A 


Fig. 4: Augmented Reality interface for gesture based programming. The user 
specifies the details of a given action through pointing gestures on the work- 
pieces (left). The robot program is automatically generated based on the action 
specification (right). 


To achieve the precision required for programming the artisan process, the 
system relies on 3D computer vision (stereo vision based on 2 RGB cameras with 
a pixel resolution of 35um/px). In an intermediate step the system than extracts 
relevant geometric features such as edges and planes. These are subsequently 
fed into an inference engine which determines probable process affordances[16] 
(areas of interest for a given process) based on pre-defined rules for the according 
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process. Using the detected affordances, the user interaction is elevated to a level 
of mutual understanding. The system presents possible actions to the user who 
than solves ambiguities and further details the action through pointing. 

Figure 4 shows the programming process for a welding process. The two work 
pieces are placed on the table. The system automatically detects the common 
edge between the objects (as it presents seam affordance) and highlights it using 
the projection system. The user than specifies part of the edge for welding using 
his finger. The system combines the knowledge about the process affordance with 
the gesture input to compute a precise robot program (limited by the accuracy 
of the 3D object detection algorithm rather than the gesture tracking). 


5 Conclusion 


In this paper we introduced the application requirements of flexible manufac- 
turing, corresponding programming requirements, and how those requirements 
can be met with two different user interfaces. Programming for flexible man- 
ufacturing introduces challenges as it requires the engagement of a number of 
stakeholders, with varying expertise and responsibilities. Two user interfaces are 
presented and analyzed according to their ability to address the various require- 
ments of flexible manufacturing. 

In future work, these two interfaces can be combined in new ways to of- 
fer new programming modalities - e.g. allowing re-programming of an action 
through gesture demonstrations. The unified testbed allows for future usability 
experiments which can quantitatively motivate best-practices for programming 
in flexible manufacturing. 
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Abstract. A central part of a robot programming system is its user 
interface. Oftentimes, its design and development are neglected, although 
the user interface is the connection between the robot and the user. In 
this work we present the application of a user-centered design process to 
a robot programming system interface. The iterative process consists of 
three steps, namely analysis, design, and evaluation that may be repeated 
until a desired outcome is obtained. The evaluation shows that with the 
help of the adapted user-centered design process, it is possible to improve 
existing user interfaces to become more intuitive and easy to use for non- 
experts from domains of assembling and construction. 


Keywords: interface design, graphical user interface, usability engineer- 
ing, intuitive robot programming 


1 Introduction 


Usually, programming robots is done by experts by writing source code. This 
approach is time-consuming and costly and therefore robots do not play a big 
role in small and medium sized enterprises. To reduce the cost and time needed 
to program a robot system, a fast and intuitive programming system that can 
be used by non-experts with little to no programming experience is needed. In 
[1] we described our fast playback-programming system that is developed for 
non-experts. Programming the robots with this system is done by manually 
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guiding them along the desired path. Afterwards, the recorded robot program 
is displayed along a timeline in the user interface. The target group for this 
programming system consists of non-experts that usually work in domains of 
assembling and construction. However, the playback-programming system still 
needs to be improved in terms of usability and intuitiveness. Therefore this 
work shows an approach towards how to design an intuitive user interface for 
touch screens and how to evaluate it. Touch screens tend to allow more intuitive 
interactions than keyboard-mouse combos [2]. Our contribution is the process of 
how an interface for an intuitive robot programming system that is controlled 
with a touch screen may be designed. At the beginning we describe our iterative 
three-stage design process consisting of analysis, design, and evaluation, where 
the changes to the existing robot programming system are shown. After that we 
show the application of the design process using the example of playback robot 
programming. Finally we show the outcomes of the evaluation steps and how 
the user interface improved compared to the previous version. 


2 Related Work 


To get an overview on the field of designing intuitive robot programming systems, 
we have to take a look at two fields of study. The first part deals with designing 
interfaces in general, and the second part shows current research in intuitive 
robot programming systems. 

Design is usually an iterative process. This approach can be found in many 
process models. Four approaches to the design process were selected as examples. 
All of these approaches relate to a user-centered design process. The individual 
phases differ greatly in terms of formulations and details. A uniform picture is 
not recognizable at first glance. The design process as it appears in the German 
DIN EN ISO 9241-210 standard contains the phases plan the user-centered de- 
sign process, understand and specify context of use, specify user requirements, 
produce design solutions, evaluate designs against requirements, and design so- 
lution meets user requirements. The design process according to [3] consists of 
the phases analysis, design, evaluation, and revision. The design process accord- 
ing to [4] contains the phases analyze, design, develop, evaluate, and implement. 
The Design Thinking process according to [5] contains the phases understand, 
observe, point of view, ideate, prototype, and test. 


Sere 


Fig. 1. Approach to a design process for robot programming systems. 
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All these design processes can be generalized to an iterative, three-step design 
approach that can be used to improve the interface design of a robot program- 
ming system. These steps are shown in Figure 1 and are called analysis, design, 
evaluation. This work does not deal with the implementation step. 

After taking a look at existing design processes, we have to look at existing 
intuitive robot programming systems to specialize our design process on robot 
programming systems. Recent research is trying to make programming robots as 
easy as possible by utilizing methods that do not require the users to write source 
code. There are multiple approaches that try to achieve this. One approach is to 
use the Programming by Demonstration approach that is either guiding the robot 
manually through its task [6] or that is learning the robot motion by observing 
the user with sensors [7]. Other intuitive approaches are to utilize hand held 
devices [8], smartphones [9], touch screen [10], visual programming in CAD [11], 
or virtual reality [12] to program robots. 

All these approaches have in common that they need some kind of interface 
to exchange information with the user. Most of the times, a screen is used to 
interact with the robot, but the design process itself for the development of an 
intuitive user interface is not shown. Therefore, a need for a design process for 
robot programming user interfaces can be identified. With the next sections, we 
want to fill this gap by introducing our design process for robot programming 
user interfaces. 


3 Design Process for Robot Programming Systems 


The research aspect of this paper is to take the proven contents of a user-centered 
design process and apply them exploratively to the design of an interface for a 
robot programming system. The goal is to design an intuitively operable inter- 
face. For this purpose, the state of the art will be examined in relation to the 
design process for interfaces. The three phases analysis, design, and evaluation 
are studied and selected methods are transferred to the robot programming sys- 
tem. The choice of methods is based on the approach of using intuition and 
personal experience to obtain an optimal result[13]. 


3.1 Analysis 


The analysis phase is the first step in the user-centered development of inter- 
faces. In the analysis, the interaction partners, man and machine, are examined. 
The investigation of the user behavior gives a complete picture of the work tasks, 
user groups, and working environment [14, p. 39]. One approach of the analysis 
is the division into goal definition, basic idea, and context of use [15, pp. 177 ff]. 
Methods of analysis include field observation, identification of personas, struc- 
tured, and free interviews [14, pp. 47 ff.]. In the intuitive robot programming 
system, the context of use consists of the operator, the robot, and the object for 
handling. In addition, the basics of the theory of intuitiveness are researched in 
order to work out valid methods within the process. 
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3.2 Design 


The design phase deals with the transfer of requirements from the analysis phase 
to an interaction platform [14, p. 65]. One approach is to divide it into different 
aspects: screen design, information design, interface design, interaction design, 
animation design, and sound design [16, pp. 12 ff.]. Methods for designing in- 
terfaces can be divided into sketching and prototyping. Methods of sketching 
are manual sketches, computer-generated layouts, storyboards, and wireframes. 
Prototypes can be paper prototypes, video prototypes, Wizard-of-Oz prototypes 
and physical prototypes [17, pp. 115 ff.]. 

Here, the implementation of the design has the goal to develop an intuitively 
operable interface. For this purpose, a paper prototype and later an interactive 
virtual mockup is generated. 


3.3 Evaluation 


The evaluation phase defines test criteria with verifiable parameters for spe- 
cific tasks. The interface can only be assessed in predefined contexts of use and 
depending on specific user types [18]. Methods of evaluation are divided into 
quantitative and qualitative methods, heuristic evaluation, and general usability 
investigation methods [16, pp. 315 ff]. The prototypes here will be evaluated with 
methods that measure the intuitiveness of the interface. Since the researched 
evaluation methods measure the usability of the interface, an adaptation of the 
evaluation method is necessary here. The MINERIC Toolkit is designed to eval- 
uate robotic systems regarding intuitiveness [19]. 


3.4 Summary 


Existing design processes aim to develop an interface that improves its usability. 
The research object of the robot programming interface intends to develop an 
interface that improves its intuitiveness. The approach is to use an already de- 
veloped theory of intuitiveness in the context of human-technology interaction 
and to explore methods that design an intuitive interface within the adapted 
design process. 


4 Example: A Playback Programming System 
After describing the design process in detail, we verify it by showing its execution 


on an example. In this example, we want to improve the already existing user 
interface for playback programming [1] in two iterations. 


4.1 First Iteration 


The first iteration contains the phases analysis, design, and evaluation. In the 
analysis phase, a research on the state of the art is carried out. This includes 
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the research on programming systems, standards, and on scientific literature to 
analyze the theory of intuitiveness. In the design phase, insights are incorporated 
into the interface design, which results in a paper prototype. The paper prototype 
is evaluated with three usability experts in an expert review. 


Analysis In the analysis phase, various systems from the field of video and 
audio editing are tested. Systems with and without touch screen functions are 
used to analyze how elements of a recording timeline can be edited in the systems. 
The topic of intuitiveness in the context of human-technology interaction was 
examined by a working group in 2006 [20]. 24 experts for man-machine systems 
were asked how intuitive work is related to the seven principles of dialogue 
design, as described in the standard DIN EN ISO 9241-110. The similarity of 
the principles in connection with intuitiveness was inquired [21]. 


Design The interface is oriented towards 
word processing software (e.g., Microsoft 
Word, Open Office Writer). It is designed in 
accordance with the standard DIN EN ISO 
9241-110. Several dialog principles have 
been incorporated into the design. Confor- 
mity with user expectations was adopted 
by implementing well-known conventions in 
the toolbar and icons that correspond to 
the functions in word processing software. 
Self-descriptiveness has been implemented 
by functions that are communicated non- 
verbally, only icons are shown on the but- 
tons of the toolbar. Suitability for the task Fig.2. Design of the paper proto- 
was adopted by active buttons that are type. 

clearly visible and buttons, like Paste, that 

appear only when a section of a recorded 

robot program along a timeline has been copied to the clipboard. 

A paper prototype is created for evaluation, which is shown in Figure 2. First 
modifications are made to the original system for keyboard-mouse combo. The 
arrangement of the buttons on the toolbar is included. For loops and branch 
functions, the additional dialog window is replaced by buttons. 

The interaction is based on the operation of word processing software with 
keyboard-mouse combo. In word processing software, a word is marked by click- 
ing on it and dragging the mouse along the word. In the paper prototype, a part 
of the robot program along the timeline can be marked by touching it with the 
finger and moving the finger along the recorded timeline. When a section of a 
robot program is highlighted, the edit functions become active. The interface 
consists of a toolbar with functions. The functions are divided into file manage- 
ment functions (new file, open file, save file), undo/redo functions, edit functions 
(copy, paste, cut, delete), and special functions (loops, branches)[1]. 
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When recording a new robot program for a timeline, the interface changes 
to the recording mode and the robot has to be programmed by manual guiding. 
The open gripper button is only active when the gripper of the robot is closed 
and the close gripper button only is active when the gripper of the robot is open. 
The resulting paper prototype is shown in Figure 2. 


Evaluation The evaluation is conducted as an expert review. The two methods 
Cognitive Walkthrough [22] and Heuristic Evaluation [23] were used for this. It 
was done with three usability experts who also do this job in their profession. The 
paper prototype from the design phase was evaluated in aspects of intuitiveness. 
It was done individually with each expert. First, in a Cognitive Walkthrough, 
the expert was guided through the individual functions of the system. Three 
tasks to be solved were described. At the end, every expert had to complete the 
questionnaire for the heuristic evaluation. 


4.2 Second Iteration 


The second iteration again contains of the phases analysis, design, and evalu- 
ation. In the analysis phase, further research on standards and design laws is 
done. In the design phase, insights are incorporated into the interface design, 
which result in an interactive virtual mockup. The MINERIC toolkit is used to 
evaluate the virtual mockup with 20 workers from the assembling domain{19]. 


Analysis In the analysis phase, more guidelines from standards and design laws 
are used. The design of a virtual mockup requires a closer examination of the 
coloring, shaping, and composition of individual interface elements. Here, the 
standard DIN EN ISO 9241-110 serves again as a support to meet important 
criteria of usability. 

Design laws play a role in the design of intuitive user interfaces outside the 
ISO standards [21]. Several design laws can be found in the literature [15]. The 
law of similarity states that related elements are recognized by similarities. The 
law of proximity states that spatially close elements are recognized as belonging 
together. 


Design The suggestions for improvement, which were written down in the ex- 
pert review, have been incorporated into the design. The interface has also been 
designed in the second iteration according to DIN EN ISO 9241-110 and design 
laws of the analysis step. 

Conformity with user expectations has been further developed by changing 
the vocabulary, so that terms from information technology (e.g. capture stimulus) 
have been changed to everyday terms (capture image). Self-descriptiveness is 
adopted by designing the record button especially present when a recording 
is expected. Suitability for the task is implemented by designing all buttons 
consistently inactive, if the function is not available at the time. In the record 
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Fig. 3. Design of the interactive virtual mockup. 


mode, the standard interface is heavily dimmed. Law of proximity is taken up 
by clustering buttons in the toolbar to show that they belong to different topics. 
Law of similarity is taken up by icons for open gripper and close gripper that 
look similar in record mode, so they can be recognized as belonging together in 
the topic, but they differ in colors so they can be distinguished. The interactive 


virtual mockup is shown in Figure 3. 


Evaluation During the second iteration of the 
design process, we conducted a user study accord- 
ing to [19]. The study took place on the 23rd 
and 24th September 2019 with 20 participants. All 
participants were workers from the assembling do- 
main aged 21 to 57 with a mean age of M = 38.7 
years and a standard deviation SD = 9.7 years. 
Ahead of the study, each participant saw a short 
video of the programming system and got an in- 
troduction into the interface. Then, each of the 
three different tasks were explained to the partici- 
pants. During the study, the participants had to 
demonstrate the tasks to a scaled, non-automated 
robot model with the help of the designed inter- 
active virtual mockup. The first task was a pick- 
and-place task with post editing where two red 
cylinders had to be picked up from a conveyor 
belt and deposited in the corresponding box, so 
that both, the recording and the editing of robot 
programs can be evaluated. The second task was a 
sensor-based repetition, where objects had to be 
picked up from the conveyor belt and deposited 
in the corresponding box, as long as the objects 
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Fig. 4. Schematic representa- 
tion of the three tasks the par- 
ticipants of the user study had 
to complete. 


were red cylinders. The third task was a sorting task where objects had to be 
picked up from a conveyor belt and had to be sorted according to their color. A 
schematic representation of the tasks is shown in Figure 4. 
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For each of the two iterations of the design process, an evaluation phase has been 
performed. The first time, an evaluation with usability experts was conducted, 
and the second time an evaluation with non-experts from the domains of of 
assembling and construction was conducted. 


5.1 Evaluation with Usability Experts 


In a first evaluation, we used a Cognitive Walkthrough in combination with a 
Heuristic Evaluation to identify possible improvement points with the help of 
usability experts. The main comment was, that the usage of the system needs to 
be consistent. For some functions, sections had to be marked first and then the 
function was triggered, for other functions it was vice-versa. Another comment 
was, that the used vocabulary should be adjusted to the target group, so it is 
understandable for them. The last major comment from the experts was, that 
functions that are expected to be used next should be placed more prominently 
within the graphical user interface. With the help of these major comments and 
some minor notes, the interactive virtual mockup was designed. 


5.2 Evaluation with Domain Experts 


The purpose of this evaluation was to get an intuitiveness rating of the improved 
interface from non-experts, namely workers from the assembling domain. Since 
intuitiveness is defined as a combination of effectivity, efficiency, and satisfaction 
according to the MINERIC toolkit, we used the presented approach to evaluate 
the prototypes. The detailed evaluation process is described in [19]. 

The effectivity score is shown in the left chart of Figure 5. It is measured on 
a scale from 0 and 100. The rating for the designed mockup is M = 69. This 
indicates a rather effective to use system. Furthermore, the score for each task 
shows that the effectivity is continously increasing, which indicates that there is 
a learning effect and it is easier to use the mockup with each task. 

The efficiency score is shown in the middle chart of Figure 5. Its value ranges 
from 0 to 220, where lower scores indicate a better efficiency. The overall rating 
of M = 39.2 is directly translatable to a little effort to use the interface. This 
means that it is quite easy to use the designed mockup. A slight increase of the 
efficiency over the three tasks may be explained with the fact that the last two 
tasks contained sensor-based loops and branches that are harder to understand. 

The satisfaction score is shown in form of the QUESI score and its subscores 
in the right chart of Figure 5. It is measured with five subscores on a scale from 1 
to 5, where 5 means that the participants are fully satisfied with the system and 
1 means that they are not satisfied. An overall QUESI score of 3.7 shows that 
the participants were rather satisfied. The subscores of the QUESI score are 
subjective mental workload (W), perceived achievement of goals (G), perceived 
effort of learning (L), familiarity (F), and perceived error rate (E). The score of 
4.2 for the perceived achievement of goals shows that the participants were able 
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Fig. 5. Outcomes of the user study. The left chart shows the results of the effectivity. 
Higher values are better. The middle chart shows the results of the efficiency. Lower 
values are better. The right chart shows the QUESI scores and subscores representing 
the satisfaction. Higher values are better. 


to achieve their goals, whereas a score of 3.4 for the familiarity shows that the 
participants were familiar with the interface, but there is room for improvements. 

Overall, the three evaluated scores, namely the effectivity, the efficiency and 
the satisfaction show that with the help of the design process, the interface for 
the playback programming system was improved and is usable by non-experts 
working in assembling and construction domains. 


6 Conclusion 


To sum up, we applied a three-step user-centered design process to an interface 
for a robot programming system. With the example application of the design 
process to improve an existing user interface in the field of playback program- 
ming, we have shown that the user-centered design process results in an intuitive 
user interface that is usable by non-experts from domains of assembling and con- 
struction. Limitations are the early stage of research of the theory of intuitiveness 
for human-robot interaction and the use of a scaled, non-automated robot model 
for the evaluation. Future work may be done by also taking the implementation 
phase into account when designing the user interface. 
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Abstract. Borescopes are a widely used technology for optically in- 
specting machinery from the inside. Recent solutions include enhanced 
metrology techniques for acquiring image data in high resolution as well 
as 3D surface information. These already foster more detailed health and 
damage inspections. However, both inspection and diagnosis strongly 
rely on the expertise of professionals. Despite their strong coupling, an 
increasing separation between these tasks is trending. Furthermore, many 
regions often cannot be inspected properly due to limitations in accessi- 
bility and hardware, which is particularly true within toroidal cavities. 
As a consequence, significant demands regarding emerging digitalization 
and automatization such as reproducibility, completeness or the steady 
and high quality of results cannot be fully satisfied. These drawbacks 
can be faced with a defined probe guidance while providing adjustabil- 
ity towards sensor characteristics. In this publication a novel approach 
with great automation capability is presented, facing the contactless and 
complete inspection of toroidal cavities, requiring no disassembly. The 
proposed solution is related to continuum robotics while based on tube 
forming technology coupled with a path planning strategy. 


Keywords: borescope inspection, endoscopy, continuum robotics, path 
generation, tube forming, cavity, probe guidance, material compound 


1 Introduction 


Increasing complexity of machinery being used worldwide, demands for nominal 
downtimes and the availability of sophisticated, non-destructive measurement 
equipment have led to more frequent routine inspections using borescopes. Re- 
cent developments have pushed these devices to smaller scales while making 
them more powerful by adding new and more accurate measurement principles. 
The areas of interest inside of machines are usually those where significant wear 
is expected due to moving parts, flowing liquids or chemical reactions. By in- 
specting these cavities regularly, damages that are invisible from the outside can 
be identified and their growth being monitored. Possible damages can include 
cracks, deformations, missing material or parts, burn throughs, abrasion, cor- 
rosion or changes in color. Typically, flexible or rigid borescopes are inserted 
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manually through available gaps, openings or bores in order to investigate the 
inner health condition of the machine. Hereby, inspection intervals can be fur- 
ther extended and a possible disassembly may be reasonably postponed. 

Dedicated professionals need to be available at the site to carry out diagnoses 
and inspections together, since each diagnosis highly relies on the subjective and 
variant way of conducting the corresponding inspection. Therefore, special ex- 
pertise is mandatory for both aspects. For the sake of more valuable results and 
an improved overall efficiency a separation of these two tasks promises poten- 
tial. Like that, inspections and diagnoses can be fulfilled independently from 
each other and thus, experts are only required for the latter. This demands for 
an inspection approach being invariant of the employee, promising a higher ca- 
pacity of inspections and the conduction of remote diagnoses. 

In some cases, especially found unfavorable when inspecting toroidal cavities, 
the borescope may be subject to gravity. This, in combination with limited steer- 
ing and controlling abilities through the inner spaces, can make it challenging 
satisfying the high standards or inspecting these parts at all. Hence, a generic 
and more robust approach is needed to help overcome these limitations. 

The novel solution proposed in this work aims for a defined and yet contact- 
less probe guidance on a circular path, being adjustable towards certain sensor 
characteristics. The path is described by the distal end of a material compound 
tube, which is formed in a certain manner. The overall system is related to con- 
tinuum robotics and is particularly suitable for inspecting toroidal cavities with 
hard to reach areas. In contrast to known high-precision solutions [1] this highly 
automatable and minimally invasive system requires no disassembly of parts. 

Next, related work is reviewed followed by a description of the proposed solu- 
tion and an experimental investigation of its capabilities. Certain characteristics 
of the system are then identified and possible limitations discussed. This paper 
is concluded with required future work. 


2 Related Work 


In this section an overview of recent developments in the field of endoscopy is 
given followed by selected continuum robots that can be used for inspection 
purposes and tube forming as the means on which this work is based on. 


2.1 Endoscopy 


An endoscope, as shown in Figure la, mainly consists of a flexible hose or a 
rigid tube. Devices being used in the technical sector are typically referred to 
as borescopes. The hose can hold different kinds of conductors, optical fibers or 
tethers and is very resistive against environmental impacts as well as mechani- 
cal stresses. Endoscopes are available in various diameters and lengths. Special 
optics for imaging as well as for lighting are located at the distal end of the hose. 
Conventionally, optical fibers are used for lighting and in the form of bundles for 
imaging also. A respective light source is then located inside of a control device 
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which also offers a representation of the image on a screen. Typically, the probe 
directly contains multiple sensor types for imaging and other purposes. Hence, 
no optical fibers are needed anymore and several electrical conductors can be 
used. The distal end can mostly be actuated in order to manipulate the direc- 
tion the head is facing. This can be controlled using tethers via the control device. 
Recent devices offer intuitive user interfaces simplifying the navigation through 
the part of interest as well as offering numerous aids for measuring and docu- 
menting features of interest. High resolution imaging sensors, which can also 
be arranged in pairs to allow for stereo vision, provide the detail of informa- 
tion needed. Recent publications describe advanced localization and real time 
tracking techniques based on these. Examples include the use of optical models 
for improvements in accuracy of endoscopic measurements [8], photogrammetry 
for obtaining 3D maps on a larger scale [3], [4] or approaches for localizing the 
probe relatively against the inner spaces using triangulation and other optical 
solutions [11] or even using shape sensing of the hose itself [14]. Additionally, 
some authors have claimed applications in microscopy using endoscopes [9]. 


2.2 Continuum Robotics 


In contrast to conventional industrial robots respective systems in continuum 
robotics usually consist of joints in high quantities that are chained together 
(Figure 1b). Beyond the capability of industrial robots of moving the flange 
within any quantity of poses these systems additionally allow for keeping the 
robots structure itself close to a defined trajectory in space. This enables for 
reaching out to inaccessible areas while still being able of avoiding obstacles. 
Such robots exist in various magnitudes of scale while especially those from the 
fields of endoscopic robotics as well as surgical applications are of greatest rele- 
vance to the objective addressed in this work. 

Endoscopic robots are designed to fit through small hollow spaces while be- 
ing required to describe winding curves. This is typically achieved by involving 
numerous individual joints. These may be actuated using conventional tethers, 
compressed or decompressed fluids, electromagnetically by coils or local motors 
in combination with gear boxes in the joints or even using piezoelectric elements 
for instance [12]. Possible configurations may be divided into those whose shape 
evolves with them being fed into the inner spaces of interest and after that can- 
not be changed anymore and more complex systems that are capable of freely 
transforming their shape at any time. The latter are known as snake robots [5]. 

The field of soft robotics comprises mechanisms that don’t rely on rigid joints 
but on flexible ones rather. These often attempt to imitate natural organisms and 
take advantage of various physical phenomena. These may be pressure, thermo- 
elastic behavior, friction, electromagnetism or the use of tethers for instance. 
Advantages are adaptability to and conservation of their respective environment, 
enhanced safety of operation in the presence of humans or smaller magnitudes 
of scale of the mechanisms being possible. 

Robots being combinations of both fields exists such as concentric tube robots 
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for instance [6]. These actually do not consist of distinct joints but allow for fol- 
lowing certain curves due to pre-bent concentric tube segments. Their constitu- 
tion is not rigid but flexible. These robots do not suffice the requirements within 
this application, therefore tube forming is introduced in the next subsection as 
a means for achieving the task. 


2.3 Tube Forming 


Various tube bending forming techniques have been developed along with the 
arising respective demands for strong and lightweight bended tubes in different 
industries such as aviation, aerospace, automotive and more. Hence, certain man- 
ufacturing parameters have improved and further developed comprising possible 
bending radii, bending angles and shapes, material characteristics and manu- 
facturing tolerances. Tube forming can be conducted either cold or warm and 
under different loading conditions like multipoint bending, compression bend- 
ing, stretch bending, roll bending, rotary draw bending or laser bending. Differ- 
ent tube materials can be processed and different basic cross-sectional shapes of 
tubes be used such as circular, rectangular or even arbitrary profiles. Fabrication 
can occur seamlessly or welded for instance. Depending on the chosen process 
parameters and the wrought material highly diverse magnitudes of forces can 
be involved in the fabrication process. Tube forming introduces some challenges 
though like possible defects in form of wrinkling, over thinning, cross section 
distortion or spring back due to complex and uneven tension and stress distri- 
butions as well as special requirements to the tools. A robust prediction and 
control of the phenomena is a common topic of research by using experimental, 
physical, analytical or numerical approaches [13]. 

Most commonly used for obtaining complex multi dimensional geometries is 
roll push bending (Figure 1c), where continuously applied mechanisms like feed 
and torsion are used in combination with certain displacements of the rolls and 
furthermore investigations on process models exist [2], [7], [10]. This technology 
is also available fully computer controlled. 

Each forming process constitutes of plastic and elastic deformation behavior 
of the material used. The latter occurs in the form of spring back of the material. 


(a) Schematic endoscope (b) Continuum mechanism (c) Roll push bending 


Fig. 1. Related work 
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3 System Description 


In the following section the novel inspection system as well as the path planning 
strategy are described and their potential investigated using an experimental 
setup. The system’s characteristics are evaluated in more detail eventually. 


3.1 System Definition 


This work aims at the generation of circular paths for the purpose of probe guid- 
ance. The underlying core mechanism is an adapted and extended kind of tube 
forming technology. By forming a special material compound tube the evolving 
trajectory described by its distal end is used to approximate the desired path 
and thus, to carry a probe. Further key components of the system include a 
guide tube (Figure 2a) and a possible actuation unit. 

The forming process may be conducted manually under certain limitations 
regarding the achievable accuracy and repeatability of the path. Incorporating 
an optional actuation unit helps enhancing the results. But still, generating the 
desired path is not trivial, unwanted deviations can occur easily and in addi- 
tion multiple curvatures are required to enter the cavity of interest successfully. 
Similar to conventional forming strategies, where both plastic and elastic de- 
formational behavior occurs, this approach makes use of both of these effects. 
Here, especially the elastic component is evoked intentionally. Along with basic 
material feed an additional and distinct portion of torsion may be used to influ- 
ence the evolving circular path’s direction and tilt. Furthermore, distinct elastic 
pre-deformations of different magnitudes and orientations can be induced using 
forming mechanisms like bending the material forth and back. 


(a) System schematic (b) Roll arrangement (c) Material compound 


Fig. 2. Proposed solution 


The entire forming strategy is organized into two stages. Abovementioned 
complex forming mechanisms are conducted outside of the toroidal cavity to be 
inspected. An additional, subsequent shaping of the material takes place using 
a guide tube where also the majority of the circularity is induced. Prior to the 
path generation, the guide is inserted and positioned through one of the possi- 
ble inspection openings. The guide tube’s shape follows certain geometric rules. 
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Clearances between the material compound tube and the guide tube’s inner sur- 
faces allow the previously applied forming steps to influence the shape, unfolding 
at the guide tube’s outlet inside of the cavity. The guide tube furthermore al- 
lows for gaining access to hollow spaces that can only hardly be reached, through 
small openings or bores, for instance. 

The material compound tube used here (Figure 2c) consists of a thin alu- 
minum strip wrapped longitudinally, thus forming the shape of a tube. The 
aluminum strip is laminated with a protective foil on either side and is adhe- 
sively encapsulated by a jacket of polyethylene. Even though the material was 
originally developed for shielding telephone cables and preventing those from 
corrosion due to moisture !, its layer structure leads to certain material charac- 
teristics that are beneficial to this work. These do not only include its capability 
of carrying cables or tethers inside but also its low specific weight per length, its 
high robustness, its highly repeatable plastic deformability and its high dimen- 
sional stability when bending curves. These especially allow for the contactless 
and self-supporting realization of the generated path while carrying a probe at 
its distal end. 

By adjusting the forming parameters, the guide tube’s geometry and the ma- 
terial’s dimensions paths of different magnitudes of scale may be realized and 
thus the method be used for inspecting a variety of parts. Beyond the use of a 
basic material and process model, additional localization techniques may be used 
to further improve accuracies and compensate deviations like sag, for instance. 


3.2 Path Planning 


The path planning strategy feeds the two-stage forming process while taking dif- 
ferent boundary conditions into account regarding the part, the sensor, the tube 
material and the forming capabilities. Figure 3 illustrates the path planning 
strategy. As an initial step, a geometric representation of the desired circular 


geometric circular circular path forming parameter tube measurement localization 
path representation segmentation computation 5 forming u sequence y technique 


Fig. 3. Path Planning Process 


circular 
path 


path is generated with respect to the toroidal cavity’s 3D model facilitating an 
optimization and equalization of the distances between the sensor and the part’s 
inner surfaces. Subsequently, the geometric representation of the circular path is 
partitioned into segments. The size of these equally sized segments is calculated 
based on the sensor’s characteristics such as its aperture angle or the desired 
measurement density, for instance. The forming will therefore be conducted in- 
termittently. Next, the different forming mechanisms such as feed, torsion and 


1 Jachimowicz, L.: Corrosion proof shielding tape for shielding telephone cables. US 
Patent 3,233,036 (1963). 
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bending needed for the first segment are quantified in order to achieve a certain 
shape unfolding at the guide tube’s outlet with the distal end approximating the 
desired path as close as possible. The forming can immediately be carried out 
either manually or using a programmable actuation unit. The forming may be 
followed by several measurements using the probe at the approached location. 
After the completion of the measurements the path planning process returns 
to the computation of the next individual segment’s forming parameters. By 
incorporating an optional localization technique as an additional part of the 
measurement sequence at each individual location a pose estimation may be im- 
plemented allowing for the closed loop control of deviation residues that may not 
be compensated using the open loop path generation. During the entire process 
the distal end’s trajectory is crucial indeed, but not the evolving shape itself. 


3.3 Experimental Setup 


An experimental setup is used to evaluate the proposed system. At first, the 
material compound tube is fed through an adjustable roll arrangement (Figure 
2b) which is mounted on a rigid frame (Figure 4a). Like this, parameters can be 
tuned in order to reach the targeted shape and thus distal end trajectory. The 
roll arrangement is further used to derive and validate a suitable geometry of 
the guide tube and therefore miniaturization of the system. Possible future guide 
tubes are also tested in form of additively manufactured specimen of plastic or 
metal. An industrial robot serves as a programmable actuation unit and is able 
to handle the tube material using a gripper and allows to continuously monitor 
process forces using a load cell. The robot conducts the intermittent forming 
using the mechanisms described above. 


= 


= s 
(a) Setup with rolls and industrial robot (b) Lasertracker plot 


Fig. 4. Experimental setup and circular trajectory plot 


During the experiments it has been found that several influencing factors for 
the resulting curvatures exist. These do not only incorporate the plastic defor- 
mation due to the forming process but also effects in form of elastic spring back 
after exiting the guide tube, gravitation having an effect on the tube’s own mass, 
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conductors running inside or a connected probe, intended pre-deformations or 
inhomogeneity of the tube’s layer structure. 

Experiments have been conducted for gaining knowledge about realizable 
diameters of the circular path, control possibilities regarding diameter and tra- 
jectory deviations, load capabilities regarding probes or conductors, repeatability 
and reversibility of deformations, forces required for the forming work and the 
system’s oscillation behavior. 


3.4 Results and System Characteristics 


Using the tested material compound tube’s diameter of 8mm circular paths of 
up to 1m in diameter could be formed, while an idealized circular shape could 
still be approximated. Accuracies can be reached down to a few centimeters. 
This was evaluated in detail using a laser tracker device. The plot (Figure 4b) 
shows the obtained path, which differs from the idealized circle with proceeding 
length, due to bending, caused by the tube’s mass. In addition to its own weight 
the system is able to carry small loads of up to 50g at its distal end as well 
as conductors inside of the tube. The forming mechanisms mentioned in the 
previous subsections can be used to control the circular path within some certain 
boundaries and to compensate for deviations, if identified. The tests revealed 
that the chosen material compound tube can be formed several dozens of times 
after which it tends to break. Even in case of fatigue no residues of the tube will 
remain since the polyethylene jacket still holds the compound together well. The 
system is prone to oscillations easily which originate from the forming process 
itself but also from external vibrations and from loads applied to the tube. 
Forming process loads were up to 80 N when pushing the material, but strongly 
depend on the roll arrangement, also. As it can be seen in Figure 4b, fast load 
application leads to significant oscillations (from 180° on), while smooth load 
application allows to avoid these. The optimized guide tubes derived from the 
roll arrangements allow for entering parts through small openings or bores far 
smaller than 20 mm. 


4 Discussion 


Since the material compound only features a limited lifespan it needs to be a 
disposable component of the overall inspection system. The circular diameters 
that can be obtained from the forming process using the guide tube are only 
representative for the tested material diameter. Other magnitudes of scale are 
possible using different material sizes. Since the overall concept is self-supporting 
and thus aims at contactless path generation with respect to the part and fur- 
thermore no residues will remain inside in case of fatigue the system may also 
be used for inspecting sensitive areas. Making use of the forming mechanisms 
mentioned above with the intention of controlling the path would imply having 
information about the current pose available. While stationary operation is al- 
ready proven, mobile operation at different sites may still be developed. Hereto, 


213 


reducing the process forces would be crucial in order to be able to miniaturize 
possible mobile forming units. Using drive units the forming can be conducted 
synchronously with possible ongoing measurements, also. Loads like probes may 
be carried by the system but care must be taken of the oscillations the system is 
prone to. This may also be done with help of the forming strategy. On the other 
hand side this fact formulates certain requirements with respect to the sensors as 
well. The chances to miniaturize the roll setup in form of the guide tube enables 
for insertion of the system through tiny openings or bores. Special about the 
suggested solution is the fact that the forming is conducted bipartite. Like this, 
hard to reach inner spaces can be accessed and a circular path be unfolded. 


5 Conclusion 


5.1 Summary 


In this work a performant new approach for the contactless inspection of toroidal 
cavities using circular path generation is presented, especially characterized by 
its high automation capability. It fosters the separation of inspection and di- 
agnosis tasks while eliminating several drawbacks of conventional inspections. 
The approach enables simplified access to inner spaces, that are hard to inspect. 
Intermittently generating distinct circular paths allows for systematic inspec- 
tion while meeting the boundary conditions of possible probes as well as parts 
of interest, leading to more valuable and reliable results. The resulting probe 
guidance system is self-supporting and therefore clearances between sensors and 
surfaces to be inspected can be well adjusted for. The solution promises to be 
configurable for various parts of interest. A first evaluation of the novel solution 
has been conducted experimentally while its capabilities and limitations were 
pointed out. Using this systematic approach completeness and reproducibility of 
measurements can be significantly improved. 


5.2 Future Work 


Certain actions need to be taken in order to further develop the proposed solution 
towards an applicable condition. These include: 


1. Material and process models need to be derived describing the material 
compound tube as well as the forming mechanisms in order to plan and 
conduct the circular path generation at improved accuracy and robustness. 

2. Localization techniques need to be evaluated to enhance the pose esti- 
mation of the distal end. Its trajectory inside of the part can then be further 
controlled by altering the path’s resulting diameter and compensating for 
deviations using mechanisms like pre-deformations of consecutive segments. 

3. A mobile drive unit is needed to utilize the solution at different sites 
of interest and perform the inspection automatically. Further research and 
optimization of the forming strategy may lead to a reduction of the required 
degrees of freedom, while the current setup is using a 6-DOF industrial robot. 
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Furthermore, an optimization of the forming process parameters needs to be con- 
ducted in order to reduce oscillations for instance. Using the advanced knowledge 
obtained by respective models of the system a design strategy for adapting the 
system for various inspection subjects is to be derived. 
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Abstract. 

In the context of Industry 4.0 and cyber-physical production systems, the role of 
production logistics is perceived as more and more important in order to reach 
the overall manufacturing targets. One central aspect in organizing the flow of 
material consists in task allocation and path planning for transport resources dis- 
posing of growing autonomy. There are various approaches for multi-agent path 
planning as well as the way of dealing with collisions. Collisions are possible due 
to traffic volume and can either be treated on planning level or in a short-term 
way on control level. 

The paper presents existing strategies for path finding before giving an overview 
of methods to deal with autonomous transport resources that meet in a manufac- 
turing environment. Then, different existing behaviors and reactions in the case 
of collision detection based on several criteria are compared. This step allows 
classifying the strategies depending on the manufacturing environment and its 
organization. 


Keywords: production logistics, autonomous transport resources, collision 
avoidance. 


1 Introduction 


With the introduction of the concepts of Industry 4.0 like digitization and decentraliza- 
tion, various challenges arise in production environments. In addition, customer de- 
mand regarding increasing number of product variants as well as shorter product life 
cycles result in smaller lot sizes. [1, 2] Production logistics play an important role in 
the manufacturing process and companies understand more and more its influence on 
the throughput time [3]. 

The utilization of autonomous resources is supposed to turn logistics more efficient 
due to their possibility to organize themselves on a decentral level. This means that the 
operational level obtains decision-making autonomy within defined constraints. A chal- 
lenge in this context is the balance between taking advantage of the autonomy and 
reaching central planning targets. One central aspect is the interaction of various re- 
sources with possibly different skill sets. [4-6] 
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Mobile autonomous resources, i.e. self-directed vehicles also called autonomous 
guided vehicles (AGV), take center stage in current research. There exist different con- 
cepts for multi-agent path finding (MAPF) presenting each advantages and disad- 
vantages. [7, 8] Considering the range of concepts for MAPF and especially collision 
avoidance an evaluation of these approaches comparing their applicability in a specific 
manufacturing environment is missing. An approach is needed in order to identify a 
suitable strategy or set of strategies for avoiding collisions in production environments. 

The scope of this paper is to give an overview of existing approaches for dealing 
with and avoiding collisions of autonomous transport resources. In addition, these ways 
are compared based on a choice of valuable criteria. 


2 State of the art 


Relevant literature concerning production logistics, autonomous resources and MAPF 
is summed up in this section to present applicable concepts. Before taking a deeper look 
on path planning and collision avoidance, a short introduction to logistics and mobile 
resources in general is given. 


2.1 Logistics and mobile resources 


As described in the introduction, AGVs are used for a variety of single and multi-agent 
applications but the main sector is logistic systems. The most common logistic scenar- 
ios are the transportation of dock containers in a terminal [9-11] and transportation 
services in warehouses as well as production lines [12]. While in the past, AGVs were 
only used for transportation services recent projects have also focused on expanding 
their skill set. Hoff and Sarker [13] have distinguished between six different major types 
of AGVs that include unit load, towing, parallel truck, forklift, light load and assembly 
line vehicles. The newest addition to these classic AGV systems have been drones 
which are tested in warehouses [14]. The vehicle design and fleet combination is just 
one of many central research topics in AGV Systems. Others involve the sensors, nav- 
igation, localization, vehicle control and the communication between the agents [15]. 


2.2 Path planning for multiple resources 


The classical MAPF problem for a set of n agents receives an input tuple (G,s,t) where 
G = (V,E) is an undirected graph with V vertices denoting locations and E edges. s - 
[1,...,.n] - V maps an agent to a start vertex , and t : /1,...,k] - V maps an agent to a target 
vertex. Often the graph is given with its corresponding adjacency matrix that displays 
the connected vertices and the weight of the edges (c.f. figure 1). For the solution of 
this problem, one can find methods, which take different constraints into consideration. 

The Dijkstra’s algorithm is one of the most commonly known algorithms for the 
shortest path search [16]. It calculates the shortest path between two vertices in an un- 
directed grid. 
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Fig. 1. Representation of an undirected graph with its adjacency matrix. 


An evolution of Dijkstra’s algorithm is the A*Algorithm, which estimates the dis- 
tance, i.e. costs, from the start vertex to the target vertex to preselect appropriate verti- 
ces with a heuristic [17]. Warren [18] proposes a modified A* Algorithm which is faster 
than the normal A*Algorithm due to a further reduced solution quantity.. Both the 
Dijkstra and A*Algorithm are not flexible and cannot react to a changing environment 
[19]. Therefor the D*Algorithm (Dynamic A*) was developed by Stentz [19] as an 
extension of the A*Algorithm. Koenig and Likhachev [20] propose the D*Lite Algo- 
rithm which combines the heuristic approach of the A*Algorithm with the incremental 
one of the D*Algorithm. Other extensions of the D* Algorithm are the Focussed D* and 
Field D*Algorithm. The Focussed D*Algorithm focusses on the repair of the cost map, 
thus reducing the computational costs [21]. While all of these approaches are limited 
by a discrete set of possible transitions in the grid the Field D*Algorithm is interpola- 
tion based and calculates smooth paths for the agents [22]. 

These algorithms were developed for single agent path planning but can also be used 
for MAPF problems. The M*Algorithm is specified for MAPF problems and minimizes 
the global cumulative cost function of all agents [23]. An extension of the M*Algorithm 
is the M*UM*Algorithm which considers uncertainty in the position and pose of the 
agent [24, 25]. Nebel et al. [26] propose an M*Algorithm for uncertainty in the desti- 
nation of the agents path while Ma et al. [27] developed a MAPF algorithm with delay 
probabilities. 


2.3 Collision detection and avoidance 


One of the main topics within the MAPF problem is the collision detection and avoid- 
ance between the agents. There are two main approaches for the collision detection: 
one is a centralized and the other one a decentralized. For the centralized approach the 
information about the actual position, path and speed must be collected in a central 
location [28]. Felner et al. [29] are adding a heuristic to the Conflict-Based Search 
(CBS) by aggregating collisions among the agents. While this approach uses discretized 
time steps and assumes that all actions have the same time duration Andreychuk et al. 
[30] propose a Continuous Time Conflict-Based Search (CCBS). 

The decentralized approach enables the agents to detect collisions by themselves and 
react accordingly. Wei et al. [31] present an approach where only the affected agents 
need to communicate with each other in case of a collision. Trodden and Richards [32] 
use a Model Predictive Control (MPC) where only one agent replans, while all others 


220 


continue with their path. The agent that replans its path is identified via a bidding sys- 
tem, where the highest bidder receives the replanning token in each iteration. Desaraju 
and How [33] use a two-step approach where the first step is the individual component 
and the second step is the interaction component. While in the first step each agent 
minimizes the cost of its own paths, the interaction component minimizes the total cost 
across all agents. 

If a collision is detected, researchers propose a variety of strategies for the behavior 
of the involved resources. Liu et al. [34] plan paths for a multi-agent pickup and deli- 
very problem where no paths cross each other during the duration of the exercised task. 
A similar approach is used by Kiarostami et al. [35] where a Monte-Carlo Tree Search 
is used to find non-overlapping paths. 

Sun et al. [36] developed a decoupled approach where every agent’s path is planned 
with an A*Algorithm. Every agent continuously checks its front area and if another 
agent is detected in a critical region, a behavior-based collision avoidance strategy is 
executed. The behaviors are specialized for an intralogistic problem description and 
selected by specified traffic rules. Another decentralized approach where possible col- 
lisions are detected by the agent itself is proposed by Gochev et al [37]. Instead of a 
tule based collision avoidance the agents switch in a collision avoidance mode. In this 
mode the agents create a roundabout along the original path which is faster than step- 
ping back and changing directions. Chang et al. [38] are using a detection shell around 
each agent and according to the nearest object inside this detection shell breaking and 
gyroscopic forces are calculated. 

A five layer neural network with three hidden layers which is able to generate cross- 
ing paths and the associated velocity vectors of the agents so that no collision occurs is 
presented by Chen et al [39]. This approach is based on the optimal reciprocal collision 
avoidance (ORCA) which is often used for the collision avoidance in free navigating 
agents [40]. 


3 Concept 


Based on the approaches presented in chapter 2, criteria to be considered in manufac- 
turing systems are identified before giving a comparison of the concepts for collision 
avoidance. For the use of autonomous transport resources in real manufacturing envi- 
ronments, this is a necessary step in order to choose appropriate strategies depending 
on individual production constraints. 


3.1 Criteria to compare collision avoidance strategies 


To evaluate the different collision avoidance strategies it is necessary to define the en- 
vironmental conditions. The criteria to compare the different collision avoidance strat- 
egies are not to be confused with the environmental settings. Since there exists no past 
work on the ranking or comparison of these influences for the different algorithms we 
have identified the following environmental influences which were the most common 
research topics in chapter 2.2: 
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e Order Duration: One of the most researched and discussed problem for the MAPF 
sector is the uncertainty in the order duration. It is not feasible to assume that the 
supposed order duration is equal to the real executed order duration because of de- 
lays that can occur. Obstructions can occur because of obstacles in the path or delays 
at the charging and discharging processes, which lead to different execution times. 
The uncertainty is not only limited to the duration of a delay but also on the occur- 
rence of a delay during the order. 

e Time Constraints: After the planning of each order, every task has a starting time 
and a deadline when it should be finished. For the AGVs, this means that it must 
reach its target vertex prior its planned deadline. Since there exists uncertainty in the 
delay of orders, the planned time for the task of the AGV must have a buffer. 

e Prioritization: Most Companies prioritize their orders because of customer relation- 
ships, demands or order volume. For this, they use different principles like First in 
First out, Least Production or scalar as weights to classify each order. 


All of these aspects have to be considered not in an isolated test area, but in real man- 
ufacturing conditions, the previously mentioned environmental settings. These com- 
prise the next parameters based on Adam [41]: 


Product information including volume and lot sizes 

Production organization including layout, workflow and in-house production depth 
Organization of planning and control processes 

Organization of logistic processes 

Presence of workers including their skills and schedules 


3.2 Comparison of different collision avoidance strategies 


There exist many different approaches for AGVs to behave in the case a collision is 
detected. We have identified five basic strategies from the research done in chapter 2.3 
and the collision avoidance behavior presented by Sun et al. [36]. Though they explain 
eight different behaviors, all of them can be reduced to two basic strategies. That is why 
we adapted the avoiding and waiting strategies of Sun et al. plus the variation of the 
start time, no crossing paths and a change of velocities for our evaluation. All strategies 
are further described in the following segment. To avoid repetitions it is assumed that 
the paths of the agents cross at vertex n, the velocities of the agents are v and the dis- 
tance from the start vertex to vertex n is d. 


1. Path Crossing: Probably the oldest strategy in case a collision is detected consists in 
finding another path, which traverses no previously planned path. This approach 
seems only suitable if there is a grid big enough to reach the target through different 
sets of vertices and edges. 

2. Priority: The agents take over the different priorities from the orders and if they col- 
lide at vertex n the higher prioritized agent passes the vertex n first while the other 
agent waits at the previous vertex n-/. In this scenario, it is possible that the waiting 
agent produces a deadlock or has to wait for a long time because of higher prioritized 
agents. The approach is shown in figure 2. 
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Fig. 2. Two crossing paths where the agents have different priorities. agent 1 passes vertex n 
before agent 2 because of a higher priority. 


3. Start time: Another possible strategy is to vary the start time of the task to avoid a 
detected collision. For each agent’s path the speed v and distance d is known, so it is 
possible to calculate the time when the agent reaches vertex n. With this information, 
the starting time of the specific order can be varied so the agent passes vertex n 
before or after the other agent reaches vertex n. 

4. Velocities: Based on the computed time to reach vertex n the velocity v can be var- 
ied, so that the AGV passes the vertex n earlier or later to avoid a collision. This 
approach can be applied in two versions. The velocity of the AGV can be set at a 
fixed value at the start of each order or the velocity can be changed temporarily 
during travelling the path. 

5. Collision avoidance mode: If one agent detects another agent or obstacle on its path, 
one or both agents can switch into the collision avoidance mode. In this mode, the 
agents pass the obstacle on their own without the need to replan. For this, all agents 
must be equipped with the necessary sensors to do such a maneuver. 


4 Application and evaluation settings 


The environment for the test cases is a ROS based path planner and its simulation en- 
vironment. A test scenario is set up to evaluate different criteria for the collision avoid- 
ance strategies presented in chapter 3.2. The parameters mentioned in chapter 3.1 are 
meant to be analyzed and structured regarding their influence on the choice of suitable 
collision avoidance strategies. They make the simulation better comparable to real life 
applications but also computational expensive. 

Workers are seen as obstacles and these will cross and obstruct the planned paths at 
a certain percentage. The duration of the obstruction is determined with a Gaussian 
normal distribution. In addition, we assume that the workstations have different states, 
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so the vertices of the grid are changing. Vertices will have the capacity one, so only one 
agent can occupy each vertex at a specific time step. The edges also have a capacity of 
one. Despite this, it is possible that multiple agents are in the same cross or pass each 
other in a lane because the ROS path planner builds its own grid on the prescribed map. 

To evaluate and compare the different strategies we analyze a set of key performance 
indicators in the simulation. Since most researchers try to minimize the sum of costs of 
all agents this is one of the identified parameters. By analyzing additional parameters, 
the influences of the different collision avoidance strategies can be shown. Further- 
more, we analyze the traffic which is generated from the different strategies and also 
the grid capacity utilization. With these parameters, we can make a first assessment of 
how the applied strategies influence workers or how easily additional orders can be 
implemented in the plan: 


e Sum ofall costs: Y_, qi where q; is the path cost of agent i and / is the quantity of 
all planned agents. The cost of each path is equal to its length. 

e Grid capacity utilization: Ratio between all the occupied vertices and edges to all 
available vertices and edges in the grid. This value shows how much space of the 
grid is occupied by agents or the agents’ path with the different strategies. 

e Traffic: Ratio between the actual duration of an order and the planned time of an 
order. With a higher ratio, more interruptions occur which equals more traffic. 


The first tests in a simulation environment compare the average throughput time of 
four AGVs with each executing orders of two or three jobs for the strategies priority 
and path crossing (cf. chapter 3.2). The results for the mean value of the order duration 
show that avoiding collisions by circumnavigating already planned paths leads, as ex- 
pected, to 10 up to 30 % longer throughput times. It is evident that these strategies 
depend highly on the considered manufacturing layout and the existing traffic. This 
needs to be regarded in a more detailed way. 


5 Conclusion 


Current challenges linked to the concepts of Industry 4.0 affect in particular produc- 
tion logistics. This comprises among others the need of more flexible solutions for ma- 
terial transport in manufacturing environments. Thus, the use of autonomous resources 
increases as these means of transport present advantages compared to standard logistic 
solutions and as they allow relocating decisions on resource level. 

This contribution presents an approach to compare different ways of dealing with 
collisions between AGVs based on valuable criteria. The treated types of collision 
avoidance were path crossing, priorities, start time, velocities and internal collision 
avoidance mode. Relevant criteria to evaluate these strategies are order duration, time 
constraints and prioritization. 

The next step will be a more extensive evaluation, validation and verification of the 
concept presented in chapter 3 within a simulation scenario. These analyses will aim at 
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verifying the applicability of the stated criteria and complete them if necessary. Addi- 
tional varying speeds of the vehicles on their way through the production environment 
have to be implemented to make the application more realistic. 
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Abstract. In production robots are moved at maximum speed whenever possi- 
ble in order to achieve the shortest overall cycle time. This can lead to individ- 
ual waiting times, especially in interlinked production processes. These waiting 
times offer opportunities for optimization. Due to high energy prices and politi- 
cal efforts, energy efficiency has become the focus of trajectory optimization in 
recent years. Robot cells with a common intermediate circuit offer the possibility 
of energy exchange across individual axes or robots. By adapting the robot tra- 
jectories, the total power consumption of a robotic cell on the grid side can be 
significantly reduced. This paper focuses on trajectory optimization, whereby a 
detailed collision detection of individual robots is included within the analysis. 
It is shown that with collision detection energy optimization for cramped robot 
cells becomes possible and the losses in efficiency compared to the optimization 


without it are minute. 


1 Introduction 


Reducing CO% emissions through higher 
energy efficiency is one of the main goals 
of our time. Operating costs of facto- 
ries can also be reduced by lowering the 
energy consumption of its components. 
As automation advances, the number of 
robots in production is constantly increas- 
ing [22]. This has also been recognised by 
the EU. An energy roadmap [7] has been 
issued in 2014 with the goal of increasing 
energy efficiency by 27 % until 2030. The 
aim of this work is to reduce the energy 
consumption of industrial robot cells by 
adapting robot movements. The method 
is evaluated using the robot cell shown in 
Fig. 1. 

The reduction of the energy consump- 
tion of robots can either be hardware- 


Fig. 1: Investigated robot cell packing a 
pallet, with several potential collisions. 


based or software-based [5]. Examples for hardware changes are e.g. extension of the 
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DC link to several robots [27] or the installation of energy storage devices [13]. With 
these two methods recuperated braking energy from the motors is stored or used for 
other robots. This means that recuperated energy for overvoltage protection does not 
have to be converted into heat [27]. However, these methods have the disadvantage that 
additional adaptation costs are incurred per robot and that systems for installation have 
longer downtimes. 


In the case of software solutions, costs are only incurred during development, and 
the transfer to several systems is negligible. Software solutions are a geometrically or 
time sequence change to a robot trajectory [5]. These solutions are a part of trajectory 
optimizations and usually refers to one robot. In [30] a robot trajectory was decomposed 
into viapoints and the position of these were optimized. In [15] the energy absorption 
of a trajectory was optimized based on maximizing the energy exchange in the DC link. 
Another possibility is a different time sequence of robot trajectories for a given task. In 
most cases several robots are involved. One method in this field is, instead of waiting 
after fast movements, the robots movement may be slowed down. In [9] different robot 
trajectories were investigated and it was found that slowing down trajectories almost 
always saves energy. However, movements can also be shifted in time so that brak- 
ing movements and acceleration, from different robots/axis, directly overlap over each 
other. Of course this only has advantages if robots can exchange energy. 


In industry this is not possible at the moment, but by implementing a DC factory, an 
energy exchange at factory level becomes possible [27]. In this paper it is assumed that 
the robots have such a connection. The method of sliding over each other can also be 
combined with the slowing down of fast movements. In [14] such an optimization for 
a multi-axis system was presented. Losses from overvoltage protection are reduced for 
one cyclic trajectory per axis. A method for scheduled energy optimal trajectories was 
presented in [29] and great energy savings were shown for a single robot. When con- 
sidering geometric path changes or when changing the time sequence of several robot 
movements, it is important that no collisions occur. [28] took collisions into account, 
but only for a restricted area and [24] used simple bounding boxes. In [2] an iterative 
collision detection for a robot was presented, which guarantees collision freedom for 
the next pose. However, to the best knowledge of the authors, for larger linked move- 
ments of several adjacent robots with a detailed collisions detection, as shown in Fig. 1, 
no uniform method for trajectory optimization was published. 


An automated methodology to exploit braking energy recuperation though schedul- 
ing and slowing down trajectories was proposed in [12]. The proposed methodology 
accomplishes this in two levels. In the first level, the sequential control optimization 
(SCO), the overall discrete control of the system is optimized to maximize the times in 
which the acceleration and deceleration phases of individual robots beneficially overlap. 
Usually, the order of robot movements can be arranged in multiple ways, and some con- 
trol strategies offer more potential for braking energy recuperation than others. Then, 
in the second level, the detailed trajectory optimization (DTO), for each set of braking 
and acceleration phases that an optimized control strategy brought to overlap, the tra- 
jectories of the involved axes must be aligned in detail. For example, the braking phase 
of one axis can be shifted, stretched, or compressed to match the energy demand for ac- 
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celerating another axis. In this paper, we focus on the DTO, a method for the overlying 
SCO was already published in [12]. 

The SCO calculates the acceleration and deceleration phase of robots that should 
overlap to maximize the energy efficiency gain. These time points and related robot 
paths are given to the DTO. Here the robot trajectories are improved in an model-based 
offline optimization. The two internal robot control variables (start delay and compres- 
sion factor) are used as optimization parameters. This guarantees an intuitive trans- 
ferability to industrial plants. The entire process is model-based and offline, therefore 
only the newly calculated parameters must be changed in the controller. In addition, a 
collision detection is presented, which can be used as a nonlinear constraint within the 
optimization. The collision detection method was only applied to DTO, because colli- 
sion in robot cells, as the one sketched in Fig. 1, are a matter of trajectory timing. This 
collision detection routine is a two-layered process [4] to guarantee a fast and accurate 
detection. The method will be tested in the simulated robot cell shown in Fig. 1. 

In Sect. 2 the energy model used to simulate the overall energy consumption of a 
production process is described. Sect.3 builds on the model of Sect. 2 to optimize the 
overall energy consumption. The optimization approach and the applied constrains are 
outlined in Sect.3. To prevent collisions, which can occur during optimization from 
Sect. 3, an collision detection is described in Sect.4. Results are evaluated in Sect. 5 
and the paper concludes in Sect. 6. 


2 Modeling of Industrial Robots 


In this section the robot model used in optimization will be presented. Since the model 
is often invoked within the optimization, one of the main requirements, besides the 
accuracy, is a short computing time. The complete model was already presented in [12] 
but it will be described roughly for the sake of completeness. The general structure of 
the energy flow is shown in Fig. 2. 

The power flows from the main 
grid Peria to the constant con- 
sumers Ĥ gria. These are normal Toc,ı Tac, 
losses caused by fans and con- ~ J hee Pees f= i 
trollers. Besides that, the main part = Zn] Pas 
of Poria flows thought the recti- = 
fier Toc,ı into the DC interme- 
diate circuit. The DC intermedi- 
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Re 
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ering the gear, the motors gener- FR 


7 O rob3 
ate a joint torque 7 and a joint 


speed %. This drives the segments Fig.2: Detailed energy flow in the investigated 
of the robot to follow a given path robots. 
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q(t), g(t), q(t) with an additional 

weight f ext. When the robot slows 

down, the motors turn into generators and recuperate energy. This energy is converted 
to DC and can supply other robots via the DC link. If too much power flows into the DC 
bus, the DC bus voltage increases. As overvoltage protection, a braking resistor Rchp 
converts the excess energy into heat. The complete energy model of the robot can be 
written as: 


tend 


E= ö Poria(t)dt = Ff m(a(t), q(t), q(t), Fe). (1) 


For a known robot, a robot path q,9,9 and an extra weight fext can be used to calculate 
the required energy. The model is nonlinear and therefore no gradient based methods 
can be used for optimization. 

The energy consumption model is validated for multiple different KUKA indus- 
trial robots [17] and has a remaining model deviation of approx. 5 % over the whole 

spectrum of operating temperatures [8]. 


3 Optimization Strategy 


Common industry robots have six joints, but the axes movements are all synchronized 
to the slowest joint. The output of a production, therefore, depends on the slowest axis 
movement. An unnecessarily strong acceleration of the other axes has no advantage 
with regard to the quantity produced. The synchronization can, therefore, lead to en- 
ergy savings, but there is no recuperation from decelerating, because all the joints are 
slowing down at the same time. The main goal of the optimization is to make all robot 
movements equally fast/slow and prevent unnecessarily strong acceleration. The opti- 
mization problem thus consists of optimization variables for compression of the trajec- 
tory öo,,; and a start delay Ts j for a robot j to synchronize accelerating and decelerating 
from different robots. The effect of these variables on a single joint robot axis can be 
seen in Fig. 3. 


—org. trajectory 
---new trajectory 


time 


Fig. 3: Visualisation of the used optimization variables. 


The start delay 7, j for a robot 7 shifts the trajectory n -time steps into the future. 
The lower limit is zero and the upper limit is the travel time for that trajectory of the 
slowest robot Ttr krit Subtracted by the travel time Tir; of the robot j. By optimiz- 
ing multiple robots with only one trajectory each, the travel time of the slowest robot 
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Ttr,krit equals the total travel time Trot, otherwise it is the sum of the the slowest trajec- 
tory Ter, krit Of the respective | multi-robot movement. The compression variable doy, 
reduces the maximal velocity q; and acceleration q; of the trajectory: 


dnew,j = Ogg i ore, N Gnew,j = Sov, j forg j: (2) 


Ttr,j 


The travel time of the robot j changes to „=. The lower limit is 0 and the upper 
limit is 1. The optimization problem finally can be described by: 


min Egria(Tss ov) s.t. 0 < Gor < 1 (3) 
= 0 < Ts,j < Ttr,krit — Ttr,j (4) 
Urs, Sov) < Ttot (5) 
C(Ts, ov) < 1, (6) 


where two new nonlinear constraints t(-) and c(-) are introduced. The first constraint 
makes sure that the total travel is at least not slower than before the optimization. c(-) 
is a nonlinear function that returns one if a collision is detected. The collision detection 
will be described in the following section and is skipped here. With this the optimization 
problem for multiple robots and multiple trajectories is fully defined. 

The non-linear optimization problem and especially the nonlinear constraints result 
in gaps in the solution area. For that reason a particle swarm optimizer (PSO [26]) was 
chosen. The PSO does not guarantee that the global minimum will be found, but returns 
an acceptable solution in reasonable time. A more detailed view on constraints handling 
and the gaps in the solution area will follow in the next section. 

For runtime reasons it makes sense to split the optimization into several sub-problems 
and solve them individually. Sub-problems are single trajectories of each of the partic- 
ipating robots, so Ttr krit equals Tio¢. The described optimization is only the secondary 
level, so the SCO matches the trajectories with the most energy saving potential. Since 
it is already known from the SCO which trajectories of the different robots have to be 
superimposed, there is no need to optimize nı, trajectories of m robots. It is sufficient 
if only one trajectory per robot, selected by SCO, is optimized in one sub problem. 

The drawback of this method is that the success of the underlying optimization 
depends on the SCO. By dividing the optimization problem, a satisfactory solution in a 
satisfactory time is obtained, but if the SCO does not find the best matches, the solution 
is not the best possible. However, without dividing an exemplary test case with 3 robots 
and 10 trajectories each can no longer be solved on a PC in a satisfactory time. 

The fact that only one trajectory per robot is considered simplifies the constraints 
of optimization. The lower limit for the compression factor ov, ; of robot j is now 
limited by the travel time of the slowest robot in the considered subproblem. Before 
there were multiple trajectories of the same robot in a row, so there was no defined end 
of a single trajectory, because trajectories could overlap. The start and end time of the 
critical robots are known by the SCO and therefore the trajectories can be separated. The 
upper limit for the start delay now also depends on the compression factor. In contrast 
to the standard problem, again without overlap, the formulation of the constraints is 
easier. This simplifies the constraints to: 
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Ttr,j 
L < bon; <1, (7) 
Ttr,krit 
0 < Ts,j < Ter,krit — (Tiri j ovj), (8) 
druda <1: (9) 


So there is no need for a nonlinear time constraint, because only through linear con- 
straints it can be ensured that the total travel time 74,4 remains the same. The collision 
detection c(-) stays the same and will be described in the next section. 


4 Collision Detection 


Using the model from Sec. 2 the optimization strategy from Sec. 3 works well if the 
robots do not share a common workspace. However, if two or more robots share a 
workspace there is the possibility that optimized trajectories may lead to collisions even 
if non-optimized trajectories do not collide. For example, there are two robots (A and 
B) standing across each other. Robot A is moving vertically and robot B is moving hor- 
izontally, so there is a possibility that these two are colliding in the middle. Assuming 
that robot B is the critical/ slowest robot, then there are two optimization parameters: 
The compression factor (override) and trajectory start delay of robot A. The collision 
detection leads now to a divided solution space as shown in Fig. 4. This (red) barrier 
depends on the compression factor and the start delay and marks a collision. Because 
of this every generated trajectory of the optimization needs to be checked for collision 
to only consider valid solutions. 
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Fig. 4: Solution space divided by collisions of a two robot cell optimization problem 


The collision model uses CAD data of the robots as well as other objects in the cell 
and imports it as a polygon mesh model. For every sample time step the positions of 
the objects are recalculated from the robot joint angles of the current trajectory. If no 
collision is found for all sample time steps, the current parameter vector is valid and 
can be used in optimization. 

The repeating collision detection routine is, as shown in [4, 18,6], divided into a 
broad phase that approximates the objects with bounding boxes and a narrow phase 
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which determines exactly if two objects collide. The broad phase calculates candidates 
that may collide and need further investigation in the narrow phase. In the past lots of 
algorithms and solutions for use in broad phase [20, 6, 11, 19, 16] and narrow phase [21, 
23, 10, 3, 1,25] with advantages and disadvantages depending on the number of objects, 
complexity of the meshes or compactness of the objects have been developed. 

Therefore for the collision detection routine within the trajectory optimization three 
different methods for the broad phase (axis-aligned bounding box pair test (AABB) 
[6], the Sweep and Prune algorithm (S&P) [6] and an oriented bounding box pair test 
(OBB)[11]) and two for the narrow phase (V-Clip [23] and the Gilbert-Johnson—Keerthi 
separating-axis algorithm (GJK) [1]) were implemented and evaluated. The selection 
was made on the basis of speed, implementation effort and availability of literature. 

To compare the selected algorithms three runtime tests were performed. This and 
all other test were conducted in MATLAB on an 15-6400 desktop computer. In order to 
guarantee comparability with the robot cell application, three tests with one trajectory 
each time were performed for the set-up of the test cell (Fig. 1) . Table 1 shows the 
average calculation times in ms for the collision check for each case. Each test was 
repeated 200 times and the average value was obtained. Due to the small number of 
tests, the deviation is very high (up to 10%), but a generalized statements can still be 
derived: Sweep-and-Prune shows its advantages in this application-related test, because 
the system is made up of many objects that move only partially, which is the strong 
point of this algorithm. Further detailed tests have shown that GJK is faster than V-Clip 
for the narrow phase, therefore GJK and S&P were chosen as the best solutions. 

To speed up the optimization it is also possible to only examine those objects that 
can collide. With the proposed optimization strategy the robots change their trajectory 
but not their path. Two objects can only collide if the spatial paths, for a given optimiza- 
tion parameter set, cross. The collision detection approach is therefore expanded by an 
extra precomputation phase. There, in advance the objects that have the potential to col- 
lide will be determined. The precomputation phase is not exact but an approximation, 
because the reconstruction of a polygonal mesh from multiple meshes is too complex 
for a preexamination. This generates from any parameter set a point cloud for each ob- 
ject consisting of all vertices from all collision sample time steps. These point clouds 
represent the paths with all time dependencies eliminated. If two points from different 


Table 1: Speed comparison for 3 different test cases in ms. Two times 100 runs are 
averaged in each case. Case 1. is test cell (Fig. 1) with 16 objects, one trajectories per 
robot and no collision, 2. are 24 objects with collision and 3. 24 objects and no collision. 


collision detection routine test case 
broad phase narrow phase 1. 25, 8. 
S&P V-Clip 1629 242 963 
GJK 1582 237 1359 
OBB V-Clip 3469 520 2082 
GJK 3405 499 1975 
AABB V-Clip 3121 486 1858 


GJK 3125 513 1923 
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Fig. 5: Power demand and relative robot speeds for the same trajectory 


point clouds are within a detection radius r they are marked as possibly colliding and 
relevant for the collision detection routine. All other objects in this robot cell e. g. robot 
bases, fences, conveyor or similar are irrelevant, as they don’t change their position and 
can never collide with the robots. 

To integrate the collision detection into the PSO three types for restraining the non- 
linear constraints are available: absorbing walls, reflecting walls and invisible walls 
[26]. Fig. 4 shows that a divided solution space can result from the layout of the cell. 
Since the particles of the chosen PSO algorithm must be able to diverge through these 
boundaries, only the invisible walls [26] are suitable for constraints handling. 


5 Results 


Fig. 5 shows the simulation results of an example system consisting of three robots as 
displayed in Fig. 1. In Fig 5 the overall grid power demand and the DC bus power flow 
between each individual robot and the DC bus in comparison to the relative robot speed 
V.er is shown. The relative robot speed is defined as the absolute sum of all robot joint 
velocities standardised to the maximum value of the slowest axis of the slowest robot. 
The travel time of robot 2 is the longest. Therefore, robot 2 is the critical robot of this 
example system. Its trajectory parameters are not optimized and the movement does not 
change between the portrayed optimization types. Fig. 5(a) shows the non-optimized 
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Fig. 6: Comparison of the energy demand with different optimization strategies 
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case. Each robot moves from its start point to its endpoint as fast as possible. This results 
in wait times for robots 1 and 3 and high grid power demands while acceleration. 


Fig. 5(b) shows the case of optimization without considering collisions. The op- 
timization leads to slower movements of robots 1 and 3 in such a way that all robot 
trajectories are synchronized. However these trajectories lead between 1.68 s and 1.78 s 
to a collision between robot 1 and 2 and are therefore not feasible. The time of the col- 
lision is marked in red. Using the nonlinear collision detection constraint as shown in 
Fig. 5(c) this problem is solved because robot | moves faster and avoids the collision. 


Fig. 6(a) shows a comparison between the energy demands of the different optimiza- 
tion strategies. Without any kind of optimization the example system consumes 4563 J. 
An optimization without collision detection leads to an energy demand of 3983 J which 
is a saving of approximate 12.7 %. Changes to the energy demand of the example sys- 
tem through collision detection are minimal and lead to 3984 J. The optimisation results 
in Fig. 5(b) and (c) indicate that it is not as efficient to match acceleration and decelera- 
tion as it is to just compress the trajectories, because there the trajectories were simply 
stretched. This has two possible causes: First, the DC link capacity consists of three 
times the amount of a single robot. This means that more energy can be stored for a 
short time. Since this capacity is not used as an energy storage but mainly as a smooth- 
ing capacity, it will probably be smaller for industrial applications, because smaller 
means less investment costs. The second reason is, that by slowing down the trajectory 
the acceleration and deceleration was also reduced and thus also less braking energy is 
recuperated. These observations can be confirmed for a larger system consisting of 3 
robots with 39 trajectories each, as shown in Fig. 6(b). By the sub problem optimization 
the overall energy saving is approximately 9 % compared to trajectories in which each 
robots moves as fast as possible and needs to wait for the critical robot. The overall 
results can be seen in Table 2. 


Table 2: Simulation results for different types of trajectory optimisation for the test cell 
in Fig. 1 consisting of 3 robots with 39 trajectories each. 


required power of the DC link process time collision 


case inJ in % ins 
no optimization 60018 100 45.585 no 
opiimizanion withour 54461 90.74 45.585 yes 
collision detection 
opumization with 54538 90.87 45.585 no 


collision detection 
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6 Conclusion 


In this paper the second part of a novel engineering methodology for the development 
of energy-efficient production system controllers is presented. The functional principle 
and results of the first optimization level (SCO) have already been published in [12]. 
This publication presents the underlying optimization strategy (DTO) together with a 
collisions detection. The complete optimization problem was set up, with two optimiza- 
tion parameters i. e. start delay and compression factor. Based on the already presented 
SCO the optimization problem could be divided into multiple independent sub prob- 
lems. Simplification for the optimization constraints were also be derived from this di- 
vision, so that no non-linear constraints are required except for collision detection. Only 
by these simplifications it is possible to solve systems with more than 10 consecutive 
trajectories. 

The disadvantage of this division is, however, that the success of this procedure 
depends heavily on the SCO. If the SCO does not find a good solution, the result of the 
DTO is also not optimal. For small cells with few trajectories (< 10) it may make sense 
to solve the general optimization problem. 

The implemented two-step collision detection was presented, as well as a short run- 
time comparison of the different methods. In addition, a method was presented which 
will remove robots from the collision detection if they can never collide with other 
robots. All methods presented were tested on a virtual test cell ( Fig. 1). The DTO re- 
duced the energy consumption when loading the pallet by 9%. Especially due to the 
large DC-link capacity, the optimization did not match acceleration and deceleration 
phases, but slowed down the trajectories as much as possible. The collision detection 
has only marginally reduced the energy saving. With this method it is possible to opti- 
mize DC-powered robots even though the space in the cell is limited. 
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Abstract. The application of unmanned aerial vehicles (UAV) in the 
area of inspection, survey or urban logistics has become a rapidly de- 
veloping research domain. While the feasibility of material transports 
with UAVs has already been shown in the scope of different projects, 
the payload is thereby usually transferred manually into the UAV’s load 
handling device. A decisive factor for the economic usability of UAVs 
for aerial transportation, however, is a fully automated system includ- 
ing the autonomous recognition and pick-up of the cargo. We therefore 
present a solution for the automated detection, localization and grasping 
of small load carriers with UAVs. The system includes a specialized load 
handing device, a camera-based real-time tracking solution for small load 
carriers and a fusion of the global and relative position measurements to 
achieve the in-flight positioning accuracy required for the autonomous 
cargo pick-up. 


Keywords: autonomous aerial vehicles, aerial grasping, object localiza- 
tion 


1 Introduction and related work 


In recent years, UAVs have found their way into various industrial applications. 
In addition to tasks in the area of inspection or surveying, there are also various 
use cases in the field of material transport, as UAVs enable fast, highly flexible 
and direct material transport in the air. In addition to the transport of parcels 
in retail or the delivery of urgent medical products or devices, UAVs are also 
suitable for the transport of spare parts or urgently needed parts on the company 
premises or within a production hall. Due to their three-dimensional scope of 
movement UAVs are able to operate in the unused space above the production 
systems, extend the flow of materials by the third dimension and offer a highly 
flexible and fast transportation. [1] 

While different publications have already shown that autonomous UAVs can 
provide an efficient addition to existing logistic systems there are still only few 
solutions or publications describing fully functional systems for the automated 
aerial in-house transportation [2]. Existing research and pilot projects on the 
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aerial transportation with autonomous UAVs commonly do not focus on the 
autonomous cargo pick-up. [3] [4] 

To enable autonomous aerial gripping of objects, different operating princi- 
ples including suction-gripping, interlocking gripping or magnetic gripping have 
been developed during the recent years. Among the most common design are 
mechanic finger or jaw grippers as they are easy to produce, robust and can 
handle objects independently of their material and shape. However, they require 
a precise positioning of the UAV or an underactuated and flexible design to be 
able to grasp objects properly. 

To compensate the displacement between the UAV and the target object 
caused by inaccuracies during manual flight Pounds et al. utilize an underac- 
tuated four finger gripper for the reliable in-flight grasping of different objects. 
Each finger features two elastic joints to compensate the contact forces between 
gripper and object, resulting from the positional fluctuation of the UAV. [5] [6] 

Optical tracking systems such as motion capturing systems allow a precise 
tracking of UAVs with an accuracy below 1mm. The usage of such systems 
allows the application of fully actuated grippers in combination with autonomous 
drones. Qi et. al. utilize a basic single-DOF finger gripper for the successful aerial 
gripping of a cuboid [7]. 

The usage of interlocking connections for aerial grasping has been shown in 
[8] and [9]. Thereby pins or clamps are actuated by servo motors to penetrate 
the cargo objects and to attain a secure fastening of the object. 


While the automated cargo pick-up is considered a key requirement for the 
cost-efficient in-house aerial transportation, the state of the art lacks of working 
solutions in this area. We thus present a system architecture for the autonomous 
detection and grasping of small load carriers with autonomous UAVs. The system 
includes an autonomous hexarotor system equipped with a camera system to 
detect and locate the cargo objects and an appropriate load handling device to 
realize an autonomous grasping. 


2 System architecture 


To ensure an economic usage of the transportation of small load carriers with 
autonomous UAVs both the cargo pick-up and the drop-off need to be fully 
automated. Therefore, we want to achieve the following transportation scenario: 
The UAV receives a transportation task including an approximate position of 
the load carrier (e.g. the position of the stockyard) and the drop point. After 
the approach to the transmitted pick-up position, the UAV detects and locates 
the load carrier and positions itself precisely above the object to allow a reliable 
pick-up. After the pick-up using an automated load-handling device the cargo is 
transported to the transmitted destination. Subsequent to the aerial transport 
the load carrier is either precisely set down on a landing area defined by optical 
markers or simply at the transmitted drop-off coordinates. 
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As we focus on the indoor transportation we use an ultra-wide band (UWB) 
tracking system for global positioning. The system allows a tracking with 10cm 
accuracy and can cover areas of several ten-thousand square meters. When flying 
in outdoor areas this localization system can be easily replaced with tracking 
data provided by satellite-based radio-navigation systems without modifying the 
remaining system components. 


2.1 Mechatronic cargo handling device 


To ensure an automated pick-up and drop-off a mechatronic load handling de- 
vice is developed and attached to a modified version of a commercially available 
hexarotor system. The load handling device is designed to be set down vertically 
on the load carrier. To compensate inaccuracies and fluctuations in the posi- 
tioning of the UAV and to center the load carrier during the landing phase the 
system provides four lead-in chamfers (cf. Figure 4). 

To fasten the load carrier after the landing, retaining bolts are used to create 
an interlocking connection with the indentations on the outer contour (1). As 
depicted in Figure 1, the individual bolts (2) are guided by linear bearings (3) 
and actuated by servo motors (4). A servo lever with a slotted hole (5) is used to 
convert the rotary motion into a linear one. As the servos are only actuated while 
the UAV is resting on the load carrier, no additional force or momentum other 
than the friction force needs to be overcome to move the bolts. During the flight 
phases, the weight force of the load carrier presses the bolts onto the bearing 
thus creating an efficient load restraint additional to the counter-momentum of 
the motor. 


Fig. 1: Fixation of the load carrier by an interlocking connection between the undercut 
of the load carrier (1) and the retaining bolts (2). The bolts are guided by linear 
bearings (3) and actuated by a servo motor (4) and a servo lever (5) 


2.2 Object localization and tracking 


Machine learning approaches are often considered as state of the art for object 
segmentation tasks. There are various neural networks available for object seg- 
mentation such as Mask R-CNN, DeepLab or FuseNet that allow an instance 
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segmentation based on colour or combined colour and depth information [10- 
12] as well as advanced pose estimation pipelines for segmenting and grasping 
various objects [13]. Light-weight models and neural network architectures also 
allow a near real-time image analysis low-cost hardware such as the NVidia Jet- 
son Nano Board or different single board computers equipped with the Intel 
Neural Compute Stick. 

However, as machine learning approaches still require large amounts of suit- 
able and annotated training data especially when the load carriers is filled with 
different objects, we favour a solution based on color and shape segmentation. 

Our approach for the detection, segmentation and localization of the individ- 
ual load carriers is based on their known colour (blue) and shape (rectangular 
with known size). The detailed algorithm to segment and locate the objects is 
presented in detail below. 

To extract the load carrier from the image, in a first step a basic colour 
segmentation algorithm is used. Based on a tolerance range centred on the actual 
colour value of the load carrier all blue pixels are segmented and converted into 
a binary mask. As depicted in the left image in Figure 2 the resulting mask 
contains most of the load carrier’s pixels but is still affected by noise and falsely 
identified areas. 

Afterwards Gaussian smoothing is applied to the binary image to reduce 
the effect of the remaining noise that can especially occur at the edges of the 
segmented objects. The Gaussian filtering prevents the incorrect detection of the 
noise as the edge of an object during the subsequent edge detection using the 
Canny-algorithm (cf. Figure 2, centre). 

By applying topological structural analysis the contours of the remaining 
objects can be extracted. As only rectangular shapes are required, we apply 
the Douglas-Peucker-algorithm to approximate the remaining contours with the 
minimal required number of points. As an rectangle is fully described by its four 
corner points all contours with more or less points are discarded. To extract the 
rectangles actually representing a load carrier their enclosed area is analysed. As 
the intrinsic camera parameters, the size of the load carrier and the current flight 
altitude is known, the expected size of the load carrier in the camera image can 
be estimated and all rectangles that do not fulfill this criterion can be discarded. 

Next, the scalar product of the vectors defining the rectangle’s borders is 
calculated to get an estimate of the angle between them. This is used to only store 
rectangular shaped contours with corner angles of approximately 90 degrees. 
Based on this information, it can be ensured that e.g. for trapezoidal shapes, 
which also consist of only four corner points, the related contours are no longer 
stored for further processing. 

Finally, the contours are sorted by their enclosed area and only the contour 
with the biggest area is kept. This is to avoid calculating the position of smaller 
rectangles within the box resulting from the possible influence of shading or 
overexposure (cf. Figure 2, right). After the segmentation the transform between 
the camera and the computed centroid of the load carrier is determined using 
the camera’s inverted intrinsic matrix. 
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Fig. 2: Binary mask resulting from the color segmentation process (left), output image 
of the contour detection (center), detected load carrier with bounding box and centroid 
(right) 


2.3 Pose estimation and software architecture 


As the optical tracking system features a significantly lower measurement noise 
than the UWB tracking system, the provided transform is not only used as set- 
point but also used to increase the in-flight positioning accuracy of the UAV. 
This is necessary as the positioning accuracy that can be achieved by the data 
of the UWB tracking system and the UAV’s intertial measurement unit is often- 
times not sufficient to pick up the load carriers. To improve the estimation of the 
current flight position we apply the covariance intersection algorithm presented 
by Niehsen [14]. 

Thereby the relative pose between the UAV and the load carrier is trans- 
formed into global coordinates and fused with the pose provided by the UWB 
tracking system. The inverse transformation is based on the UAVs global posi- 
tion at the time of the initial localization of the load carrier. 

Using the covariance matrix Kxx of the global UWB tracking data and the 
covariance matrix Kyy of the pose provided by the optical tracking a combined 
covariance matrix K,, can be calculated as 


Kez = (w * Kx! + (1—w) * Kyy 4) 


(1) 
To find the optimal solution for K,, the value of w is computed in such a 
way that the determinant of K,, is minimized. Using the available closed-loop 
formulations a fused position estimation 


Pr = Kez * (w* Kyx | * Po + (1— w) *Kyy '* Pa) (2) 


is calculated and forwarded to the internal kalman filter of the flight control unit 
(FCU). Thereby the position values provided by the UWB tracking system and 
the box detection are denoted as Pg and Po. 


The implemented process for cargo pick-up, transportation and drop-off is 
depicted in Figure 3. Initially the overlying task management sends an approxi- 
mate position setpoint to the FCU, indicating where the cargo object is located 
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Fig. 3: Flowchart of the implemented software system used for object segmentation, 
pose estimation and controlling of the load-handling device 


(not shown in the chart). As soon as the UAV reaches the provided position the 
process for locating and grasping the load carrier becomes active. 

Based on the provided RGB-data the load carrier is segmented and localized 
using the algorithm presented in section 2.2. The resulting transform and the 
known transforms between the camera and the FCU and the load handling 
device and the FCU are then used to calculate an updated position setpoint 
that is forwarded to the FCU. 

Using the covariance intersection algorithm an improved position estimation 
is calculated whenever valid relative tracking data is available and forwarded 
to the internal kalman filter of the FCU to improve the in-flight positioning 
accuracy. 

The internal land detector of the FCU provides information whether the 
UAV is already landed or still in-flight. Based on this feedback, the load handling 
device controller drives the individual servo motors to fasten or release the cargo. 

After a successful pick-up (which is determined by the increment in the re- 
quired take-off thrust) the UAV heads for the provided drop-off position. When 
the drop-off position is referenced by additional optical markers, they are tracked 
using the algorithms presented in [15] and the resulting relative position is used 
as input to the covariance intersection algorithm, otherwise the positioning dur- 
ing the landing phase only relies on the UWB tracking data. 


3 Evaluation 


The resulting system is depicted in Figure 4. The developed load-handling de- 
vice is attached to a DJI F550 airframe with the corresponding electronic speed 
controllers and motors. As FCU we use the open-hardware system Pixracer run- 
ning the 1.9.2 stable release of the PX4 autopilot [16] and a NVidia Jetson Nano 
on-board companion computer (OBC) for object detection and data fusion. 
The implemented software utilizes the Robot Operating System (ROS) and 
the provided ROS communication layer. To allow a precise tracking of both 
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Fig. 4: Hexarotor system equipped with the developed load handling device 


the load carrier and the landing area two down-facing cameras are attached to 
the system. The first is located in the center of the mounting plate of the load 
handling device and the second one is mounted to the outer frame of the UAV. 
The individual servo motors are connected to a suitable driver and controlled 
by the OBC. 

While the FCU runs the real-time modules such as attitude and position 
controller the OBC runs additional software modules required for general flight 
processes and navigation. Core element of our previously developed software 
framework is the mission controller, a software module that is responsible for 
basic navigation tasks, path-following, system diagnosis and failure detection. 

Figure 5 compares the achieved in-flight accuracy based on the pose esti- 
mation of the internal kalman filter when hovering around 1m above the load 
carrier. To simplify the evaluation and depiction the corresponding static set- 
point is subtracted from the position values. All flights are conducted with the 
same closed-loop controller but it’s effect on the positioning accuracy and poten- 
tial improvements will not be discussed in detail. When only the UWB tracking 
information is used, the resulting in-flight position deviates around +5cm from 
the setpoint in x- and y-direction. However, the deviations in z-direction reach 
values of up to 20cm. This can be traced back to the fact that the noise in the 
measurement values of the UWB tracking system in z-direction is significantly 
higher then the noise in x- and y-direction. 


When fusing additional optical tracking information, the deviations decrease 
to maximal values of +4cm in every direction. Compared to the UWB-only 
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Fig. 5: Comparison of the achievable positioning accuracy when only relying on UWB 
tracking information and when fusing UWB and optical tracking data using the co- 
variance intersection algorithm 


dataset, especially the accuracy in z-direction improves significantly enabling a 
consistent and precise descending. Compared to the z-direction, the positioning 
accuracy in x- and y-direction increases less which can be traced back to inferior 
measurement inaccuracies of the UWB tracking system. 

While the optical tracking of the load carrier improves the flight accuracy 
and improves the success rate of the cargo pick-up further improvements are 
required. The lead-in chamfers of the current load handling device can compen- 
sate movement of up to +5cm, however, the automated pick-up currently only 
succeeds approximately every second time (evaluated on the basis of 20 test 
flights). Besides the in-flight position oscillations also faulty tracking informa- 
tion, friction on the lead-in chamfers and oscillations caused by the ground effect 
are accountable for the poor success rate. 


4 Conclusion and Outlook 


Within this paper we have presented an approach to the autonomous aerial 
grasping of small load carriers with UAVs. Using a camera-based solution for 
segmenting and locating the cargo objects, a newly developed load-handling 
device and the covariance intersection algorithm for optimized pose estimation, 
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the automated pick-up of small load carriers was realized. While it was shown 
that the autonomous pick-up of small load carriers is possible with the present 
system, the reliability and rate of successful pick-ups still needs to be improved. 

To optimize the flight and grasping performance within future research, the 
influence of optimized and model-based controllers will be examined. Also im- 
proved cameras with optimized field of views and additional sensors to measure 
the current movement speed will be integrated and the influence of vibrations 
caused by the spinning rotors will be examined. 
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Abstract. Even for modern robot attached gripping solutions, surface-sensitive 
object handling remains a challenge. A minimal gripping force is required to grip 
and handle a surface-sensitive object safely. At the same time, this gripping force 
should not exceed a certain level to avoid damaging the sensitive surface of the 
object. Because of this, it is necessary to determine the mostly small range of 
optimal gripping forces to ensure a safe handling. Therefore, this paper describes 
an approach to evaluate this range of gripping forces for surface-sensitive objects 
using by means of chocolate covered marshmallows. For this purpose, a test 
bench with the same characteristics as a planned gripper is developed. Through- 
out the project, this planned gripper will be controlled with the help of the de- 
tected range of gripping forces. In addition to these static measurements, it is 
planned, that the gripper will calculate the handling speeds, accomplished by a 
six-axis industrial robot, with its acceleration forces. The achieved results of this 
paper will be the starting point for the upcoming measurements with the robot. 
Furthermore, the influence of the temperature on the range of gripping forces was 
evaluated, to show the relevance of consistent environmental conditions for the 
handling of surface-sensitive objects. 


Keywords: Gripper, Handling, Force control. 
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1 Introduction 


In areas like the food industry, robots are often used for heavy load handling, e.g. pack- 
aging and palletising. In contrast, the handling of sensitive objects is still a major chal- 
lenge for industrial robots. Recent articles [1, 2] illustrate the need of a sophisticated 
control strategy for sensitive object handling in various industries. Furthermore, the 
acquired sensor data is often used for the handling process alone and not broadcasted 
towards a MES or CPPS for further process analysis and decision making [3]. Available 
gripping solutions usually focus on the process of gripping and grasping without taking 
translational movements during transit into account. Especially sensitive objects are 
prone to be damaged by dynamic and abrupt changes of acceleration. This can be ex- 
plained by the corresponding inertia that adds up to the applied static gripping force. 
As soon as the sum of dynamic and static forces exceeds the object’s limits, it is dam- 
aged and may no longer be suitable for its designed purpose. In order to provide a suit- 
able solution, the project Sensitive Robot Based Gripping (SenRobGrip) aims to de- 
velop a cyberphysical gripping system (CPS), which is capable of handling sensitive 
objects whilst improving the cycle time. To handle surface-sensitive objects, it is re- 
quired to know the range of optimal gripping forces to handle it safely without damag- 
ing it. In some cases, this range can be so small, that even little changes of the environ- 
ment can affect and lead to a shift of this range. The gripper should include this range 
of gripping force and the acceleration of the moving robot to optimise the pick-and- 
place task while transit. The base design of the gripper and the motion planning ap- 
proach for the robot movements is described by the authors in former publications 
within the SenRobGrip project [4, 5]. Subsequently, this paper determinates the range 
of optimal gripping forces using the example of chocolate covered marshmallows and 
the influence of temperature. These forces are necessary to adjust a later process control 
with the real gripper. For that, a test bench is developed which covers the same gripping 
characteristics as the planned cyberphysical gripping system [6]. 


2 Use Case 


The food industry provides many different practical examples for the demand of sensi- 
tive object handling. The range of products is varying in consistency, geometry and is 
often covered by pressure-sensitive surfaces, which break or show signs of maltreat- 
ment if not handled correctly. This maltreatment is directly linked to customer satisfac- 
tion and may lead to decreasing sales. In order to automate pick-and-place processes 
for products like these, it is necessary to consider static and dynamic forces within the 
handling processes. To provide a realistic use case, the developed gripper is designed 
to be able to handle chocolate covered marshmallows at high velocities and accelera- 
tions without damaging the surface. The proposed scenario makes it necessary to de- 
velop a handling device for these size variable and extremely fragile products in order 
to provide a solution for a broader product spectrum. Furthermore, environmental cir- 
cumstances like temperature and humidity have a distinct influence on the product and 
therefore have to be monitored throughout the production. 
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3 Gripper Hardware 


The first step is the development of a new and versatile gripper as shown in Figure 1. 
The design aims to provide high stiffness for an accurate gripping force measurement 
as well as for being flexible enough to handle various object geometries. The gripper 
consists of a hollow cylinder to which matching flange adapters for different robot 
models can be attached. For communication and control purposes, a Raspberry and Ar- 
duino are placed inside the cylinder and mounted to a 3D-printed inlay. The finger 
drivetrains connect to the bottom of the cylinder. To improve stiffness and reduce fric- 
tion, the fingers are attached to linear motion guides and are propelled by a belt drive. 


Fig. 1. Gripper hardware design. 


Four 24 V DC motors actuate the motion and generate a finger movement speed of up 
to 0.4 m/s which allows the gripper to close in less than 0.15 seconds.! The device is 
equipped with four exchangeable piezo-resistive force sensors — one in each finger — 
which provide precise measurements on a scale of 0 to 10 N and an accuracy of up to 
0.01 N at 860 samples per second.” In addition to this, a 6 degree of freedom (DoF) 
inertial measurement unit (IMU) MPU-6050 provides the needed information about the 
robot’s movements. Thus, the accuracy of up to 0.598 mm/s? for translational motions 
and 0.0153°/s for rotational movements at 1 kHz allows robot-autarkic control. This 
enables the manufacturer-independent usage, as no proprietary interface is needed. To 
monitor environmental influences, a dedicated sensor gathers data on environmental 
influences like temperature and humidity in order to provide a feedback to the produc- 
tion management system to maintain stable process parameters. 


' Faulhaber 2250S024BX4 CSD with 1:14 227 gearhead 
2 Honeywell FSGOLOWNPB with HX711 analog-digital-converters 
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4 Gripping Force Test Bench 


The aim of the gripping force test bench is to simplify the testing using chocolate cov- 
ered marshmallows and to define the object specific range of gripping force in this 
example. The general construction is shown in Figure 2. The chocolate covered marsh- 
mallow lies on a small podium in the centre of the device and is surrounded by four 
single grippers. Each of these grippers have its own carriage combined with a rail for 
the linear movement powered by a servo drive with the help of a belt drive. The gripper 
jaw is a 30° segment of a circle with a radius of 30 mm and a height of 10 mm made of 
Polylactide. The podium in the center can be lifted up and down by a seesaw to see if a 
chocolate covered marshmallow is held safely by the grippers immediately. 


Fig. 2: CAD model of the gripping force test bench 


The sensors for force measurement are built into these gripper jaws. To read out the 
values of the sensors and implement a controller, an analog digital converter is used 
and connected to an Arduino Due microcontroller by an I2C Bus System. 


5 Experimental Design 


The main objective of this experiment is to evaluate the range of the gripping force to 
handle chocolate covered marshmallows without damaging them. This range is limited 
by the minimal force which is needed to hold the object and the maximum force to hold 
the object without damaging it. First, the minimal force can be evaluated with the po- 
dium and the seesaw in the centre of the test bench. After the gripper is closed around 
the object, the podium can be lowered to see if the chocolate covered marshmallow 
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stays in position. In a second step, the maximum forces can be evaluated by searching 
for damages of the sensible chocolate surface. Within the following experiments, the 
surface damages are divided into five stages. Stage zero represents a not damaged sur- 
face and the other four stages describe the level of damaging from a visible light up 
pressure spot (stage 1), over one crack (stage 2) and many cracks (stage 3) to a complete 
break of the chocolate (stage 4). These four stages of damage are shown in Figure 3. 


Fig. 3: The four stages of damage 


Before explaining the evaluation and showing the first test rounds a technical problem 
needs to be explained. We identified a random effect of shaking between two opposite 
grippers. After some adjustments of the controlling and the Arduino software it was 
possible to nearly eliminate the shaking after some test runs. Afterwards it occured for 
a few cases only, but it mostly effects the minimum gripper force which is needed to 
handle the chocolate covered marshmallow. These cases will be marked in the upcom- 
ing results. The shaking results from the combination of fast high-end sensors with very 
high resolution for the measurements and simple motors providing limited step sizes to 
change the gripper jaws positions. The final gripper with the more expensive motors 
was already in production so we were not able to use these motors for the gripping force 
test bench. This leads to problems in the force controlling which is shown by the shak- 
ing sometimes. For the final gripper we will not expect these problems thanks to the 
high-quality motors that will be implemented. 


256 


To evaluate the minimum gripping force the gripper is adjusted to test in specific 
ranges. 

The forces are measured when the gripping process is finished, which means that the 
motors stopped moving and the object is gripped without the podium underneath. After 
some iterations the range for the minimum gripping force at room temperature (RT) is 
detected between 0.15 — 0.35 N based on each single gripper. In order to later include 
the movements of the robot into the calculation, each gripper finger is individually con- 
trolled. For the following 30 tests, a three-point controller with the determined range 
was tested without losing grip at any test. The results are shown in Figure 4. The four 
values of the individual grippers are combined in the graph to form an overall gripping 
force for simplification. The developed lower limit for the range of gripping force lies 
at around 0.88 N. 


Minimal Gripping Force, RT 
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Fig. 4: Measurements for the minimal gripping force at room temperature 


To define the upper limit for the range of gripping force every tested chocolate covered 
marshmallow is gripped twice to ensure that no damage is missed out in the first at- 
tempt. After some iterations the range for the maximum gripping force at room temper- 
ature is detected to range between 0.8 — 1.2 N based on each single gripper. Again, 
random force values within this previously determined range were tested. 26 chocolate 
covered marshmallows are tested twice, which results in 52 measurements. The results 
are shown in Figure 5. 


257 


Maximum Gripping Force, RT 
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Fig. 5: Measurements for the maximum gripping force at room temperature 


Two objects are damaged after the first grasp and four objects are damaged after the 
second grasp. This leads to eight negative measurements compared to 44 positive ones. 
The measurements marked with “* Stage 2” are from a marshmallow with a very rough 
and bumpy chocolate cover, which results in higher pressure peaks. This shows the 
importance of a good production quality for future handling tasks. 


The objects 20 and 25 were damaged at the second grasp. The single gripper forces 
from the four measurements are shown in Table 1. For object 20 the crack is between 
gripper 2 and 3. The cause of this crack could be the rise of the gripping force from 
1.03 to 1.08 at gripper 2 or 0.88 N to 0.97 N at gripper 3. For object 25 the crack is 
between gripper 1 and 4. In this case the rise from 1.11 N to 1.15 N at gripper 1 or 0.99 
to 1.09 at gripper 4 could be the reason for the crack. This shows the small range of 
tenths and hundredth of Newton gripping force that can lead to a crack in the chocolate. 
Because of these small fluctuations the upper limit for the overall range of gripping 
force is set to 3.5 N as shown in Figure 5. 


Tab. 1: Detailed look on two tested objects 


Single Gripper Force [N] 


Object number Measurement | Grasp | Gripper 1 | Gripper 2 | Gripper 3 | Gripper 4 


56 39 1th 0,98 1,03 0,88 1,03 
40 2nd 0,97 1,08 0,97 1,00 
49 Ith 1,11 1,00 0,99 0,99 
25 


50 2nd 1,15 1,00 0,95 1,09 
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Furthermore, the effect of temperature on the range of gripping forces is evaluated. The 
chocolate covered marshmallows are cooled down to 8°C and the procedure for finding 
the range of gripping forces is the same. As shown in Figure 6, the lower limit for the 
range of gripping forces at 8°C is higher compared to the equivalent limit at room tem- 
perature. While testing the maximum gripping forces for the upper limit of the range, 
the previously mentioned shaking of the test bench is a problem again. This shaking 
results in some gripping force peaks in Figure 7. Furthermore, some chocolate covered 
marshmallows got a crack from the temperature differences. Overall, the results shown 
in Figure 7 lead to an upper limit at around 5 N at 8°C. The different results between 
the measurements at 8°C and room temperature were expected due to the adhesive and 
melting characteristic of chocolate. 


Minimal Gripping Force, 8°C 


1,3 
Z, 12 ren Pee SS aSESs =: ereceececeecece È. = eeeceececeecececeee ®© 
S41 © e °? 
= © a e ° e 
= 1 ® ® e © ® 
o e © © ® ® o 
= 0,9 @® = © © (J 
&D r 
© 0,7 
0 5 10 15 20 25 30 35 40 
Measurement 
@ held @ minimal slipped @ slipped +++. Linear (lower limit) 
Fig. 6: Measurements for the minimal gripping force at 8°C 
Maximum gripping force, 8 °C 
9,000 
$200 e 
O 
(J (J 
7800 
fs) @e o eo © ee Ge 
a © 
6800 
F 2-09 =. 
5800 ---- u - ---525----=°-285---- 7-25 -- 
9 oo ET 9,008 IE ta 
4,000 
0 10 20 30 40 50 
Measurement 
@ StageO @ Stagel 
@ Stage2 @ = cracked from temperature 


----- Linear (upper limit) 


Fig. 7: Measurements for the maximum gripping force at 8°C 
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6 Results 


First of all, the results show that it is possible to determinate the range of gripping forces 
for surface-sensitive objects like chocolate covered marshmallows. Also, the problems 
with the shaking grippers will not be expected when using the higher quality motors in 
the final gripper. Even with the applied cheaper motors for these experiments have 
shown that the required sensitivity is possible. The evaluated range of gripping force 
for the example of chocolate covered marshmallows is found between the limits of 0,88 
N to 3,5 N at room temperature and 1,2 N to 5 N at 8°C. These results show the limits 
for the controlling of the surface-sensible gripper developed in this project. The force 
sensitive object handling needs to determine and actuate the controlling in these defined 
limits to allow a safe handling of surface-sensitive objects. It should be emphasised that 
these tests are made on a specific test bench without the acceleration forces of the robot. 
This will reduce the range of motion and emphasize the importance of controlling the 
individual fingers. We are currently trying to transfer these results to the real gripper 
(shown in Figure 8). 


Fig. 8: The real gripper 


Furthermore, the results show the importance of the temperature controlling of the 
product environment. The handling of surface-sensitive objects in winter or summer 
could make a difference in product quality ifthe season affects the ambient temperature, 
yet this paper shows that with some information about the effects of the temperature — 
and by integrating the measurement into the cyberphysical gripper control - it is even 
possible to enable a safe handling. 
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Abstract. In times of ever increasing performance requirements imposed on 
newly developed products, fiber reinforced plastics have become essential for 
lightweight applications. While mass related stiffness and strength are huge ad- 
vantages, automatic handling of the raw materials, such as fabrics, can be very 
challenging. Low-pressure suction grippers have been proven to be capable of 
handling fragile air-permeable materials in a non-destructive manner, giving 
them a clear advantage over potentially damaging competing gripping technolo- 
gies such as needle grippers. However, they also consume large amounts of en- 
ergy to ensure successful handling operations. This research aims to estimate dis- 
turbance forces applied to the manipulated workpiece by monitoring its internal 
suction pressure, which will in turn allow for running the grippers at a bare min- 
imum power while still being able to react to disturbances enabling safe handling 
operations. 


Keywords: Process Monitoring, Gripping Technology, Textile Handling. 


1 Introduction 


While weight related stiffness and strength of continuous-fiber reinforced plastics 
(FRP) are superior to most classic metallic materials [1]. There is a lack of automation 
solutions in the handling of “limp and air-permeable semi-finished textile products” 
[2]. Low pressure suction grippers have been identified as a technology with great po- 
tential for automated handling of semi-finished textile materials [3, 4]. 

They can maintain a moderate suction pressure at high volumetric flow rates [5], 
which allows for sufficient holding forces even when air is able to pass into the suction 
cavity ofthe gripper either by leaking airstreams passing the material or by penetrating 
the porous material directly. 

For automated handling of air-permeable sheet materials, low pressure suction grip- 
pers have been used in a variety of research applications [6-9]. 

However, due to a lack of compensation mechanisms for ‘disturbances during the 
handling process’ they are often run at maximum suction power to ensure reliable ma- 
terial transportation even when exterior forces may interact with the handled textiles. 
This mode of operation is very energy-inefficient and thus not desirable. [10] 


© The Editor(s) (if applicable) and The Author(s) 2020 
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This work focuses on estimation of disturbing forces applied during handling of air- 
permeable blanks by monitoring the interior pressure inside the grippers’ central cavity. 


2 Low Pressure Suction Grippers 


Grippers using partial vacuums at moderate volumetric flow rates have been identified 
as one of the most promising technologies for automated textile handling [11]. 
Fig. 1 shows a sketch for a simple suction gripper. 


Qu 


Suction cavity Fy 
Suction plate Comm 
> F G 
Fabric Fy 
Fcp 


Fig. 1. Sketch of a simple suction gripper and Forces acting on a gripped Fabric. Based on [10] 


These systems usually feature central cavities, which are partially evacuated to a low 
pressure throughout this paper the differential pressure between suction cavity and sur- 
rounding atmosphere will be depicted by py. 
A suction plate, which may contain small holes preventing major material deformations 
and restricting airflow in non-covered areas, is usually attached to the gripper. This 
avoids “short circuits”, that would lead to a rapid decline in suction pressure py [12]. 
In a one dimensional lifting operation the handled materials are generally subjected 
to four different forces [10]: 


e Fy, a holding Force exerted by the pressure drop across a gripped sheet. An equation 
formulated by Götz relates the holding force exerted on the gripped material Fy to 
the pressure-drop Ap across the material by definition of an effective area Aeff. [12] 


Fy = Aeff ` Ap (1) 


e Fg, the force excerted on the object by gravity. 

e Fy, a combination of other dynamically acting forces in normal direction. Such as 
inertial forces or external forces these will hereinafter be called disturbances or dis- 
turbance forces. 

e Fp, a constraining force applied by the suction plate contacting the fabric. 


The contact force with which any gripped material is pressed against the suction 
plate can be described by [10]: 


Fcp = Fy + Fg + Fy (2) 
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3 Evaluation of the State of the Art 


3.1 Pressure Monitoring for Suction Grippers 


Götz measured differential pressures inside a low pressure handling system for leather 
blanks during operation for validation purposes, however, he did not relate them to 
significant process properties. [9] 

The gauge pressure inside a suction Gripper has been used for gripper control by 
Ehinger. The interior pressure was controlled by a PI-controller to match predefined 
values found by experimental analysis. Allowing for secure draping of a textile material 
even with some non-covered holes on the suction plate. [6] While controlling the inte- 
rior pressure will lead to a response on disturbing forces acting on the handled material, 
these forces are not quantified. For that an additional observation of the supplied suction 
energy would be necessary. 

Fleischer et al. have shown a dependence of the internal negative pressure inside a 
coanda-effect based grippers’ suction cavity on the number of manipulated layers for 
air-permeable materials. In experiments placing up to 2 layers of textile materials be- 
neath a suction plate and gradually decreasing the input power-stream to the gripper it 
was found, that for any given input power the interior pressure approaches a higher 
vacuum with additional layers adhering to the manipulator. This allows the system to 
directly estimate the number of sheets separated from a stack-magazine. [8] 


3.2 Gripping Force Estimation for Low Pressure Suction Grippers 


Fleischer et al were able to measure holding forces between a suction gripper and car- 
bon fiber materials by attaching brass electrodes to the suction plate. The measured 
electrical resistance between the electrodes has been shown to directly depend on brass- 
carbon contact resistances, which in turn decrease with higher holding forces applied 
to the handled textile. By designing control loops for energy efficient operation, they 
were able to create a dynamic system capable of quickly reacting to disturbances, while 
also reducing energy consumption by nearly 90% compared to a conventional static 
energy supply. Furthermore, they were able to control the holding force, such that a 
reliable separation of singular sheets off of a stack-magazine was possible. [8] 

This approach, however, relies on the electrical conductivity of the handled textile 
and is therefore not applicable for a wide variety of common materials. 

Brink et al. developed a “Force Controlled Variable Fan Gripping Module” incor- 
porating an electrically powered fan as a vacuum generator. Strain gauges were inte- 
grated into the fan grill, which serves as a suction plate in this system. With increasing 
handling forces, increasing deformations in the grill can be observed, which were in 
turn registered as a change in resistance for the strain gauges. Using this principle, Brink 
et al. were able to monitor process forces occurring during the handling of non-crimp 
fabrics. They furthermore noted, that a dependence of the gripping force on the number 
of attached layers is observable. [7] 
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Retrofitting sensor integrated suction plates introduces undesired complex altera- 
tions to commercial gripping-systems. A less invasive approach for measuring disturb- 
ances acting on the gripper is required. Monitoring the pressure inside the suction cavity 
promises to be an efficient solution to this problem. This research shows an approach 
for generation of a process model relating negative pressure to disturbance forces based 
on an offline calibration. This system can be used for online monitoring of handling 
operations. It may also be used as part of a state estimation algorithm allowing the 
gripper system to control contact forces at the gripper-material interface. 


4 System Description 


The SCG 1xE100 gripper used in this research (see Fig. 2) is manufactured by J. 
Schmalz GmbH. 


Fig. 2. Left: SCG 1xE100 gripper with suction plate 
Right: cross sectional sketch of a coanda-effect based gripper based on [8] 


The gripper is activated by supplying air at a positive operating pressure py to port 1 
this high pressure air fills a primary cavity, which it can escape through a thin slit lead- 
ing to a central secondary cavity. The inwards moving air will tend to adhere to the 
slightly curved top surface. This primary flow Qy facing upwards will have a suction 
effect on the air at port 2 creating a negative pressure py and a secondary airflow Qy. 
During operation an additional suction plate is screwed onto the gripper at this interface. 
The combined airflow will leave the suction cavity by port 3. Another connection from 
the outside directly to the suction cavity is provided by port 1/2, which is usually used 
for a blow off functionality [5]. 

Initial tests probing the internal suction pressure at different positions by laterally 
inserting hypodermic needles into the suction cavity have shown, that even small trans- 
lational variations of the probing-position can have small (varying by about 0.05 kPa at 
suction pressures of py = 0.1 kPa) but significant effects on the resulting pressure 
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readings. These differences are presumably caused by the non-homogenous suction 
plate geometry and turbulent flow regimes directly behind the it. Therefore, an optimal 
probing position should be situated as far away from the plate as possible. For transfer- 
ability of measurements amongst different SCG- 1xE100 grippers the blow-off port 1/2 
has been chosen as an ideal probing position. 


5 Procedure 


An adapting gripping system capable of monitoring and reacting to changing disturb- 
ance forces may be realized by using the controller architecture shown in Fig. 8. 

In such a system a desired contact force Fcp ges can be set and controlled, which will 
directly implement dynamical reactions to any disturbance forces acting on the handled 
material (see equation (2)). A PID-controller can be used to dynamically change the 
operating pressure of a suction gripper depending on an error measurement based on 
desired and estimated contact forces between the gripper and the handled textiles. 


Desired Contact |F cp,des. Pu Pu Qn 
——— PID-Controller 
Force 
Contact Force Frest 
Fcp ase Estimation 


Fig. 3. Proposed control loop for the final gripper system. Parts analyzed in this research are 
highlighted 


This research aims to show how a disturbance force estimation in the shape of a model 
F,(Qy, Py) can be obtained by taking the following steps: 
1. Recording of example pull-of-experiment data 
2. Construction of a mathematical model based on the obtained data 
3. Qualitative confirmation of the correlation between measured and estimated 
disturbance forces by applying the model to further validation tests 


6 Experimental Setup 


In order to characterize the correlation of disturbance forces Fy and the suction pressure 
Py an experiment as shown below is set up. 

The experimental setup sketched in Fig. 3 consists of an SCG 1xE100 gripper that 
can be moved up and down by a distance s with a lead screw driven by an electric 
stepper motor (not depicted in the figure). The force F, at which the gripper is pulled 
towards the base is measured by a force sensor. The material specimen is clamped be- 
tween two metal plates with a central rectangular cutout (100x100mm seen in Fig. 3.). 
The operating pressure py is supplied by a proportioning valve, while the flow rate 
Quis measured by a flow meter and suction pressure py being monitored by a differen- 
tial pressure sensor. 
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Fig. 5. Leakage occurring after deformation of the gripped fabric 


Pull-off tests have been performed by supplying a specific operating pressure py and 
moving the gripper upwards by an increasing distance s, while recording Qy, F and py. 
This movement leads to a gradual separation of the material from the suction plate al- 
lowing for additional air to bypass the manipulated sheet, avoiding the pressure drop 
across the textile, leaking into the suction cavity and thereby decreasing the measured 
value of py. This leakage happens in the test stand as well as when gripping freely 
hanging sheets as depicted in Fig. 4. 


7 Experimental Results 


7.1 Model Construction 


This section discusses the aforementioned offline calibration for the carbon fiber mate- 
rial shown in Fig. 3. 

Exemplary pull-off tests yield the results presented in Fig. 5, with pull-off forces 
F and differential pressures py plotted against the traveled distance s (Fig. 5 a). py and 
F are plotted against one another from test start up until the maximum force Fax 1s 
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reached (Fig. 5 b). The same plot is repeated on the right for measurements after Fax 
was reached (Fig. 5 c). The clusters of lines shown in Fig. 5 correlate to supply pres- 
sures of 0.5; 1; 1.5; 2; 2.5 and 3 Bar. 
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Fig. 6. Measured forces and negative pressures for an exemplary pull-off test 


At the beginning of any given pull-off-test the pressure py stays nearly constant, with 
no noticeable separation of the gripped material from the suction plate. Recorded forces 
are here mainly dominated by the material deformation. After 0.5-1.5 mm of travel 
slight separation can be observed with a decline in py. The recorded forces keep rising 
until a maximum force Fmax is reached close to an inflection point seen in the 
Py X s plot between | to 4 mm of travel. At s = ~5 mm full separation can be observed 
with forces quickly approaching 0 N and py settling at a non-zero value generated by 
the flow resistance of the installed suction plate. 

Since the supply pressure at the gripper is set and not directly measured, the operat- 
ing air flow measurement Qy, which behaves almost proportional to py has proven to 
be more reliable in dynamic situations. As seen in Fig. 5 the negative pressure increases 
practically linearly with increasing operating pressure, while decreasing almost quad- 
ratically with increasing disturbance forces. Based on this observation a simple poly- 
nomial modeling approach py(F, Qy) = Poo + PıoF + PoıQn + Pi1F Qn + PooF? is 
chosen. A parameter optimization for the specific results shown in Fig. 5, gives a pol- 
ynomial numerical model, relating pull off forces and operating pressure to the meas- 
ured differential pressure (before Fnay was reached): 


Pa 
Pu a(F, Qu) = -1826Pa — 279.8 F r 


74.02 Qy — 56.72 F? 2 — 3.168 F Qy —— (3) 


Pa 
lmin™1 
Here, Qy is given in norm liters per minute. 


The model represented by equation (3) reaches an adjusted R? = 0.9930 over 224 sam- 
ple points collected in 18 individual experiments. 
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Analogously, another model can be obtained for the part of the tests after the maximum 
force has been reached. 


Pa 
Pup(F,Qy) = —301.3Pa — 161.5 F N + 


9.124 Qu u + 15.64 F? — 0.8202 F Qu (4) 


Nlmin-1 
Using the same approach, similarly close approximations have been found for a variety 
of different materials. > two models described above can be combined using a linear 


separator at py = -4000Pa — 600Pa. 


160 l min” 


In Qn); if — -4000Pa — 600Pa < py m 
Pu p(F, Qu) ‚else 


For any combination of py > 0 and Qy > 0 a corresponding estimated disturbance 
force F, can be found by calculating the force, which minimizes the absolute difference 
between the predicted and measured negative pressure: 


F (Pu, Qu) = argmin(|pu(F, An) — Pul) (6) 


7.2 Model Application 


Manually moving the gripper up and down on the test-stand allows for a comparison 
between measured forces and estimations made by the previously described models, 
solely dependent on py and Qy. Results of these simple validation tests at an operating 
pressure of py = 2 bar can be seen in Fig. 6. 

It is clearly visible, that significant disturbance forces and in turn separations are 
directly seen by the estimation. While both shown curves overlap for most parts of the 
diagram slight deviations can be seen, which may be explained by non-elastic behavior 
of the tested fabric. 
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Fig. 7. Model evaluation by manual movement of the gripper 
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In another test manually pulling the fabric down similar observations were made as 
shown in Fig. 7. 


Measured 
Estimated 


F in [N] 


Fig. 8. Model evaluation by manual movement of the fabric 


Measured and estimated forces closely match one another while forces are applied 
evenly around the gripper. Under this load case the model estimation shows an average 
error of about 1 N. 

With punctual loads (as seen at t = 20s in Fig. 7) forces will not be estimated as pre- 
cisely. Here, the estimated force is off by almost 100%. Nevertheless, it can be con- 
cluded, that the estimation model qualitatively shows good correlation with the meas- 
ured data. 


8 Conclusion and Outlook 


This work has introduced a differential pressure-based estimation method for disturb- 
ance forces acting on fabric being manipulated by a low-pressure suction gripper. After 
obtaining experimental data, polynomial models can be formulated which serve as a 
look-up point for the desired force estimation. It has been shown, that such estimation 
model can be used to deduce qualitative information about forces acting on the gripped 
material. While these estimates do not show perfect resemblance of the forces, espe- 
cially for different load cases, the two show a strong correlation. 

In future research these results will be generalized to allow for model estimation 
based on material properties and gripper characteristics. Which will then be used to 
define a control structure capable of adjusting the systems. As shown by the works of 
Förster such a control structure can furthermore be used for robust separation of singu- 
lar sheets of air-permeable material from a stack magazine [3], paving the way for 
highly automated production lines in both classical and technical textile industries. 
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Abstract. The change in production towards smaller batch sizes demands a high 
degree of flexibility, especially in assembly. In this context, human-robot collab- 
oration (HRC) is becoming increasingly important. The goal is not to displace 
the human being by the robot, but instead receiving support from it. Nevertheless, 
employees at all levels have to be prepared for the HRC. Trainings offered from 
robot manufacturers focus on robot specific operations and are not sufficient for 
the development of competencies for innovative and creative process planning 
and optimisation. Therefore, a mixed reality learning environment can be used 
for the implementation of such training, since real production facilities are only 
available to a limited extent for training purposes. In the context of the imple- 
mentation of a mixed reality learning environment, the question arises as to the 
necessity of haptic feedback. This article presents a concept for integrating HRC 
in German initial and continuing vocational education and training using mixed 
reality. 


Keywords: Human-Robot Collaboration, Learning Factory, Assembly. 


1 Introduction 


The importance of flexible assembly is growing in particular due to the increased pro- 
duction of small series through customer-specific productions [1]. The stepwise auto- 
mation of the production process enables a cost-effective production of customer-spe- 
cific products. In particular, the design of human-robot collaboration (HRC) plays a 
central role with regard to future assembly processes, whereby the form of cooperation 
between human and machine is decisive. The implementation of HRC enables a high 
overall productivity and a better product quality. [2] However, the safety of the em- 
ployees showed as a difficult problem [5]. Therefore, the planning of collaborative pro- 
cesses can be supported and validated in a virtual environment [6]. The virtual environ- 
ment is based on a digital twin using data from new product variants and resource con- 
figuration. It allows a simulation of the collaborative processes including the human. 
The results show, that in order to ensure a safe use of collaborative robots and increase 
their potential, qualified employees are required. [7] Therefore, employees at all job 
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levels must be qualified, because it is unlikely that HRC will result in production with- 
out employees and it does not represent a desired goal. Rather, the human being remains 
the most important control and decision-making authority. [3] The aim should be to 
place the human being at the centre of the process in the sense of designing the work 
processes [4]. 


In most cases, the system integrator trains the employees in the use of a specific 
robot. This training is limited to the teaching of programming knowledge and product- 
specific basic qualifications. Initial research, as part of the KoRA third-party funding 
project indicates that these training courses do not meet the expectations from employ- 
ees in the company. In this context virtual environments offer a safe and reconfigurable 
learning environment. GAMMIERI ET AL. already published a virtual environment for 
HRC enhancement [8]. They integrated a forward and inverse kinematic of a KUKA 
collaborative robot and detected collisions. Furthermore, the digital model can be con- 
nected with a real robot. Due to the integration of several collaborative assembly sce- 
narios, the virtual environment can be used for safety considerations as well as for train- 
ing operators. Another approach form MATSAS AND VOSNIAKOS introduced a virtual 
reality training system for HRC [9]. The System visualises safety issues like contacts 
or collisions and can be used to investigate acceptability of HRC. Moreover, OYEKAN 
ET AL. investigate the reactions from human to movements of the robot [10]. The goal 
is to test collaborative processes in advance to ensure a safe process. However, none of 
these approaches provided neither a differentiated learning environment based on the 
individual qualification level, nor an appropriate teaching concept. 


2 Mixed Reality Learning Environment - KoRA 


In September 2019, the University of Bremen started a research project named KoRA 
with the aim to prepare employees at all levels for HRC. The teaching and learning 
concept enables employees to view complex assembly systems holistically in order to 
recognise the potential of HRC and use it in a targeted manner. The approach uses a 
learning software made in Unity in combination with learn and work assignments (see 
section 3) that rely on the visualisation techniques of Mixed Reality. Within the assign- 
ments, the dissolving of rigid borders between manual and automatic assembly is fo- 
cused by the use of collaborative assembly. 

The possibility of participation of the employees offers the potential for dismantling 
obstacles to collaboration with robots [9]. This is of particular important because the 
employees conviction about the advantages of HRC is a crucial component for the suc- 
cessful introduction into the factory [11]. The complex assembly scenarios are prepared 
within the framework of learning and work assignments. These take into account oper- 
ational reality and real work processes and thus enable project-based learning in realis- 
tic scenarios. In particular, scenarios on the topics safety, collaboration suitability and 
technical feasibility are intended (see Fig. 1). The concept addresses trainees, skilled 
employees and supervisors, as well as trainers and teachers in relation to the German 
dual vocational training. 
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Fig. 1. Concept of the KoRA mixed reality training 


3 Didactical concept 


The didactical concept is based on learn and work assignments (see Fig. 2). They are 
particularly suitable for dual vocational training due to their occupational world rele- 
vance, the link to problematic situations and the project-based learning organisation 
[12]. Especially, learners benefit from the problems they face in their daily work be- 
cause the relevance, i. e. the present and future significance (according to KLAFK)), be- 
comes apparent to them [13]. 

The learn and work assignments are also characterised by their project form, which 
promotes the independent development of a problem solution, both as an individual and 
as a team effort. The acquired knowledge is applied in the development of solutions, 
the planning of the work steps as well as the implementation and adaption to the indi- 
vidual starting position. 

A further advantage of learn and work assignments is the option of flexible design. 
Heterogeneous target groups identified to be taken into account and to be trained in a 
targeted manner. The learn and work assignments can be differentiated for the different 
modes of the learning application and the corresponding learning objectives as well as 
competence focal points. However, the focus of the gradation is not on the qualification 
level, but rather on the necessary characteristics of the competence profile for the em- 
ployees in the collaborative robot environment in the individual company context. The 
different focal points are magnified in the KoRA learning software in different modes, 
which have to be selected by the user at the beginning. 
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Fig. 2. Principle of learn and work assignments (according to HOWE [12]) 


Besides the employee, the trainer, in school, company and inter-company context, 
represent another important target group for the concept. They are addressed in train- 
the-trainer workshops and understood as a central element of the didactic formulation 
of objectives. In addition to the didactic concept, the trainers are equipped with pre- 
pared content and methods for their own teaching and learning context. This lowers the 
inhibition threshold for the use of the learning software and supports a use beyond the 
duration of the project. 


4 EMMA - Experimental modular assembly plant 


The experimental modular assembly plant (EMMA) of the bime Institute consists of 
six autonomous sub-modules, which can be arranged flexibly (see Fig. 3). Three mod- 
ules are designed for manual assembly processes and two are fully automated. A col- 
laborative module is used to supports another manual assembly station. All assembly 
stations are connected by a belt conveyor for transportation. The manual stations also 
have a large touch screen for interactive communication with the employee. This al- 
lows, for example, a support for the assembly process to be displayed. As an example 
product, a ball labyrinth is produced on the plant. This ball labyrinth has been designed 
in such a way, that it can be dismantled non-destructively. The assembly of the product 
requires the use of various joining processes, from clips to screws to the knotting of a 
rope. Together with a variety of possible variations of the product, this results in varia- 
ble assembly scenarios. The plant as a whole is available as a virtual image and supports 
a cyber-physical linking between real system and VR environment. 
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Fig. 3. The experimental modular assembly plant (EMMA) 


The collaborative module can be used for several assembly tasks during the production 
of the ball labyrinth. One possibility is the support during the rack assembly. Within 
this task, the robot positions the profiles in such a way that the employee can carry out 
the required screw connections in an optimised manner. The robot is programmed of- 
fline in advance on the basis of CAD data. In practical operation, however, collisions 
can occur between robots and employees. This can be caused, for example by individual 
use of the workstation or by the size of the employee. In this case, a hand-guided teach- 
in of the robot by the employee is necessary in order to adapt the movements of the 
robot to the respective situation. With the intention of avoiding downtimes in produc- 
tion, the adjustment process must be carried out efficiently by the employee. This sce- 
nario can be used as an initial scenario within the virtual learning environment for first 
investigations about haptic feedback. 


5 Haptic Feedback within the learning environment 


For a Mixed Reality learning software it is necessary to enable a realistic representation 
of the virtual environment. Furthermore, the applicability of the hardware solution at 
the different learning locations is a requirement for the application and its technical 
implementation. Therefore, the current technological developments and the resulting 
possibilities and limits have to be considered. The integration of a haptic feedback for 
the user increases the individual immersion into the scene [14]. This aspect has already 
been tested in a number of scientific studies. In case of virtual reality training in man- 
ufacturing a multi-modal feedback enable effective transfer of virtual acquired 
knowledge to real-world [9]. However, a corresponding hardware for the consumer 
market is only available in limited cases. The consumer hardware is regularly limited 
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to feedback gloves and trackers for tracking physical objects. For the use in the educa- 
tional system, the simple purchase and a clear integration effort are decisive. The pro- 
ject investigates the necessity and influence of haptic feedback for and on the learning 
process in a virtual environment. Therefore, a stepwise integration of haptic feedback 
for different tool classes into the learning software is planned (see Fig. 4). A distinction 
is made between the following tool classes: static tools (e.g. screwdriver), dynamic 
tools (e.g. electric screwdriver) and kinematic tools (e.g. Cobot). Moreover, aspects 
such as human perception, cognitive signal processing and individual visual abilities 
are taken into account. 


Learn and Work Assignments in KoRA 


Computer System 


> 


Sequence 


LWA (>) 


Visualisation 


EEE ei 


: p e :  Head-Mounted 


3D-Data 


Tracking i anaana a 
Haptic 

Feedback 
Device 


© bime, Sv-0300 


Fig. 4. Experimental setup to evaluate the required haptic feedback 


Regardless, it must be investigated under which conditions a higher immersion leads 
to a higher learning success. This relationship has been investigated particularly in the 
medical field, where the use of mixed reality is already frequently used in doctor train- 
ing. RANGARAJAN ET AL. have analysed nine studies where a haptic feedback is used in 
surgical mixed reality education [15]. They report that six out of the nine studies come 
to the conclusion that a haptic feedback significantly enhance the simulation and re- 
duces the learning curve. Two studies shows no difference and one study suggested 
negative effects for training due to decreased realism. RANGARAJAN ET AL. conclude 
that there can be no general recommendation for the use of haptic feedback. Rather, it 
must be decided individually for each scenario whether a haptic feedback contributes 
to the learning success and the additional benefit justifies the costs. [15] 
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Within the framework of the KoRA project, extensive studies are therefore carried 
out to determine the required degree of haptic feedback. For this purpose, a first test 
scenario will be created based on Unity and different levels of haptic feedback will be 
integrated. On the one hand, physical components will be tracked and integrated into 
the virtual environment and on the other hand, feedback gloves will be used to simulate 
contact situations. In order to examine the haptic feedback that is required, the test sce- 
nario is first carried out with vocational school classes. The degree of feedback is varied 
and the learning curve of the participants will be observed. Afterwards, the test will be 
repeated with employees from companies. Based on these results, the decision of which 
haptic feedback justifies the associated effort can be evaluated. 


6 Summary 


This paper describes a concept for a Mixed Reality learning environment that transfers 
the approach of collaborative robotics into the German initial and continuing vocational 
education and training. Initial studies indicate that particularly in this area exists a gap 
in actual teaching and training concepts. The introduced concept is based on the didac- 
tical concept of learning and work assignments, which represent project-based and em- 
ployment-oriented learning. All scenarios within the learning software use an experi- 
mental modular assembly plant, which exists as a virtual environment. This allows a 
stepwise integration of haptic feedback elements to analyse the increase of the learning 
curve. Afterwards, a decision for each scenario has to be made between the benefits 
and efforts of feedback elements. 
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Abstract. Modern aircraft engines require appropriate care and understanding of 
design and manufacturing. This is even more important, as the production of aer- 
ospace engines remains a manual process in many cases with limited data 
sources. Quality control has to consider verification of manufacturing and assem- 
bly steps through specific checks and controls whilst implementing additional 
data sources. 

In this article, a review of the challenges with regard to controls, automation 
and process, and technical understanding for aerospace engine production and 
repair is provided. As this requires the collaboration of many teams and partners, 
an improvement and step change towards deeper understanding and process effi- 
ciency is required. As many operations remain manual, innovations in the way 
humans interact with technology and collaborate with industrial environments 
are needed. 

This article demonstrates the creation and usage of the proposed solutions for 
collaboration, troubleshooting and error correction in jet engine assembly. 


Keywords: Industry 4.0, Autonomous Robot Programming, Collaborative Cog- 
nitive Robots, Augmented Reality, Virtual Reality. 


1 Introduction 


If transportation is a blood-vascular system of our world, then all motions elements of 
this are red blood cells. The systematic prophylactic is needed. 

The area of the jet aircraft is a high frontier of the new technologies. It is jet aviation 
that allows people to travel the fastest traveling. Unlike high-speed trains, the network 
of air routes covers the whole world. 

It has been proven that technological processes and quality control of aircraft engine 
maintenance are key factors for advancing the process of continuing air-worthiness 
support. [7] 

Unlike many other modern high-tech areas like car and combustion engine assembly 
lines and production lines of electric motors, the assembly and maintenance of jet en- 
gines is quite conservative. In production and in service many manual labor operations 
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are used. This is a factor of an additional risk in spite of numerous checks and controls. 
Automatic assembly (e.g. with the use of industrial robots) is not applicable. 

This makes cooperation difficult. But maintenance and repair of jet engines requires 
the coordinated work of a large number of people and factories. 

To ensure the reliability of aircraft engines, it is of great importance to develop meth- 
ods for complex multi-parameter diagnostics of technical condition. [15] 

In the research projects Cockpit 4.0 and Insight Industrial Analytics - running until 
mid-2020 - Rolls-Royce Germany and the chair of automation of the Brandenburg tech- 
nical university Cottbus-Senftenberg is already developing the first digital technologies 
in the field of assembly, logistics and the linking of development and production / as- 
sembly. 

Important is not only the use of several manual operations during the maintenance 
and repair but the use of old communications and data recovery methods during a lot 
of these operations. It seems first of all about the use of “paper documentation” in the 
working area and the problem of its actualization. 

To solve these problems, considering the restrictions of the production environment 
and the strict regulations in the aircraft industry, the following tools were chosen within 
the research project: 


1. Object Detection 

2. Collaborative Cognitive Robot (CCR) 
3. Smart Assembly Model (SAM) 

4. Smart Human Machine Interaction 


This is already the first step forward in product understanding and increased opera- 
tional efficiency. 

However, the partners expect a significant quantum leap through the far-reaching 
combination of collaborative cognitive robot, smart human-machine interaction, oper- 
ational machine-related data, logistics information and the exploitation of the so-called 
digital thread from concept, development, production / assembly, testing to service. 

The approach can be generally applied to all maintenance fields. In this research, the 
before mentioned tools are described and the way they improve manual maintenance is 
examined. 


2 Object Detection 


2.1 The Detector 


For the robotic object detector in the jet engine assembly, the faster R-CNN detector 
[17] is used in combination with the large Resnet101 network, which has a high average 
precision (more on that in the next section). Detectors are divided into region based and 
single shot detectors, where single shot detectors (SSD) have a much higher frame rate 
enabling to perform real time object detection at the cost of lower accuracy. Region 
based detectors like the mentioned faster R-CNN detector demonstrate an accuracy ad- 
vantage if real-time speed is not needed. 
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According to the elaboration of Hui from 2018, compared to other detectors, faster 
R-CNN is not the fastest but one of the most accurate detectors [10]. 


2.2 Network Architecture 


In the robotic vision application, the faster R-CNN detector sits on top of a neural net- 
work which was trained by the synthetic training data (mentioned in the next section). 
There are two methods to train a neural network — one where the network architecture 
will be assembled manually layer by layer with convolutional layers, pooling layers, 
fully-connected layers, hidden layers etc. It takes a lot of computing power and time to 
train the layers and their millions of neurons to detect low level contours, colors, shapes 
etc. in order to recognize objects. The other method is to use a pre-trained model which 
can already detect various objects like animals, furniture, humans etc. and only over- 
write the classifier layers with the objects that need to be detected. However, in this 
method, the base layers (feature extraction layers) -where the neurons already learned 
how to identify contours, colors, shapes etc.- remain pre-trained. This is giving the ad- 
vantage of a short training time and layers already trained by experts for state-of-the- 
art feature extraction. Within the research project the last approach was used. As a pre- 
trained model, the Resnet101 with the already trained coco image dataset [6] was used. 
Everything that needed to be done was the training of the top classifier layers with 
training data of objects that need to be recognized. 


2.3 Synthetic Training Data Generation 


To generate enough training data and increase the diversity of object appearance, we 
create multiple sets of rendered images. In these images, we differentiate the position, 
orientation, scale and material of the 3d object as well as the intensity of the light 
source. Two sets of 3d objects are sampled, root objects and non-root objects. For root 
objects, we create an animation sequence of 10.000 frames. The object is animated to 
rotate in Z and X axes during the frame range. This process provides images of the 
object at every angle with a 2 degrees threshold. Furthermore, is taken into considera- 
tion the pose of the object in the real world and no image is created where the object 
will not be accessible with the camera, thus, avoiding feeding the CNN with unneces- 
sary data. For non-root objects the same process is followed, and the images created 
correspond to every angle of the referred object with a threshold of 4 degrees. In order 
to improve the perception of our CNN we use material diffuse colors as close as possi- 
ble to real ones. To achieve that we use the actual camera that will be later used, to take 
photos of the object in different light conditions and different angles to receive realistic 
range of texture color. Subsequently we pick a range of RGB colors from the photos 
and use it as diffuse color in our 3d material. Throughout the animation mentioned 
above, the diffuse color of the object changes smoothly between all received colors. 
During scale and position transformations, we keep at least 25 by 25 pixels of the ob- 
ject’s image inside the safe frame of the camera. 

The parts to be inspected will not only be free in space but also parts of an assembly, 
thus the need for our CNN to be able to recognize parts fused with each other. To 
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generate appropriate data, we integrate the assembly steps into the 3d environment. 
Starting with the full assembly, we dedicate one render channel per object class and 
setting its name to the class representative. With all but the first layer of objects, ac- 
cording to fitting instructions, hidden and with regard in transparent background image 
format and alpha channel enabled we render the entire animation sequence. In each next 
step of the rendering process we unhide the next visible object class in the assembly 
and render the animation sequence. 

After completion, if the assembly consists of 10 different object classes, thus assem- 
bled in 10 steps, we have 100.000 training images. 

Furthermore, with the multi-channel rendering, each object which is visible in an 
assembly step, also has rendered images. The 550000 images of the multi-channel ren- 
derings are used only to generate annotation data in a VOC-Pascal format for each im- 
age. 

The next section describes the approach of how to automatically generate annotation 
data for our training images. 

Creating annotation data. With OpenCV the bounding boxes can be obtained for 
each object in the assembly sequence. Therefore, multi-channel images are created 
from one assembly scene. The main image of a specific assembly step has a different 
channel for each object. These multi-channel images only display the (partial) objects, 
which are not concealed by other objects. From these separated objects, OpenCV can 
detect bounding boxes of pixel groups. The coordinates of the bounding boxes will then 
be written into the annotation file. All the bounding boxes of all the objects from one 
image (main image) of an assembly step will be gathered in one annotation file. Both 
the main image and the annotation file will then be fed as training data into the neural 
network. This method allows to automatically generate training data from 3d models in 
any partial assembled state, allowing the classification layers of the neural network to 
recognize partial concealed or assembled objects. Combining the large number of syn- 
thetic training images, with a small number of real images with ground truth human 
annotations, will form a training image set with high performance [8]. 


2.4  Overfit-Resistant Image Synthesis 


We synthesize thousands of images for each 3d model. While the realistic effect con- 
sidered to be better, images with simple lighting conditions and materials prevent deep 
CNNs from picking up unreliable patterns. We simplified the creation of clutterness 
embedding all important randomizing elements together. For image rendering, we ex- 
plore parameters of rotation, scale, move and light intensity. For the position, scale and 
orientation condition, we sample values according to the original size of each object as 
for the deformed object to occupy enough space in the final image thus, to be exploited 
by the CNN. For the lighting condition, intensity of the light source is sampled. Images 
rendered as above have a fully transparent background, and the object boundaries are 
highly contrasted. To prevent classifiers from overfitting such unrealistic boundary pat- 
terns, we synthesize the background by a simple and scalable approach [19]. 

For each rendered image, we sample multiple backgrounds of solid colors, photos of 
industrial environments, nature and of same-textured-parts assemblies. We use alpha- 


289 


composition to blend a rendered image as foreground and a scene image as background 
[19]. Images cropped by cameras safe frame and images with cropped parts by other 
objects from the multi-layer renders teach the CNN to recognize occluded or truncated 
images. Finally, we use 75% of our created images as a training data, while the rest 
25% of it is used as validation data. 


3 Collaborative Cognitive Robot (CCR) 


3.1 Cognitive Vision 


With Chapter 2 we created a visual feedback, where a robot can identify objects in his 
workspace and calculate rough positions in his base coordinate system. The rough es- 
timation of an object’s position in the 3d workspace of the robot can be achieved by 
using the knowledge of the size of the detected object, the constraint that the object 
always stands on a known height (floor) relative to the robot and the xyz-coordinates 
and angles of the robotic tool center point (camera position). Since the accuracy of the 
calculated pose will be far from good and only represent a rough estimation, a second 
iteration is necessary for the robot to get the exact position of the detected object — 
getting into close contact with the haptic feedback. 


3.2 Haptic Feedback 


Haptic calibration. For a robot it is necessary to know the exact object position or 
damage might occur. Making a robot calibrate itself is great for versatile workspaces 
without fixtures, granting possible human collaborators freedom in their own ways of 
working. The CCR achieves this self-adjustment with the help of camera-based object 
detection and a force-torque sensor. Both can be attached to any off-the-shelf collabo- 
rative robot. In the experimental study a force-torque sensor (FT-sensor) from Robotiq 
(FT300) and a camera module with the google coral edge TPU dev board was used. 
The process of self-calibration involves calculating a rough position of the object with 
the camera-based object detector and finally the haptic calibration. Within the haptic 
calibration, at first the robot slowly moves close to the object’s estimated position with 
a TCP speed slower than 15cm per second. Then the robot will be close to the object 
performing a slower tactile move (Scm per second) towards the objects surface. When 
the surface is reached the force applied to the sensor should exceed 2 N or 0.12 Nm, 
which is gentle enough to not move the object if its weight is above 2 kg. The force 
threshold for recognizing the touched object should not be lower, since the FT-sensor 
has a certain noise level. When touching the first contact point of the object, two or 
more additional points are required for calculating the middle point and angle of the 
object in the base coordinate system of the robot. After the self-calibration with haptic 
feedback, the object’s position will be accurate within +/-5mm, depending on how the 
touching tip of the TCP is designed. A very small touch tip would grant the best accu- 
racy, however, since the application is collaborative according to EN ISO/TS 15066, a 
surface on the robot must never be smaller than 1mm2 resulting in a larger tip and a 
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lower accuracy. The resulting position error of +/-5mm can be challenging for follow 
up assembly or pick and place operations. To compensate the error, again the FT-sensor 
can be used for force-controlled operations. 

Force-controlled operations. If the position of an object is not programmed or cal- 
ibrated correctly, damage can be caused to involved components. To become more 
flexible and versatile a robot can be equipped with a force-torque sensor. With robotic 
skills like “spiral search” or “linear search” the robot can use haptic feedback to identify 
object positions or assembly positions simply by monitoring a force drop or rise on 
contact or successful insertion. Sometimes, the misalignment of the case makes the first 
insertion attempt fail. With the spiral search algorithm, the bushing is able to be inserted 
in a flexible way. Each time a bushing is inserted, the correct assembly position is used 
to correct misalignment or calibration errors. This way the robot learns from his assem- 
bly attempts to identify the exact position of the object. 

Combining visual with haptic feedback clears the path for flexible and versatile col- 
laboration between robots and humans while using self-calibration and force-controlled 
operations. 


4 Smart Assembly Model (SAM) and Autonomous Robot 
Programming 


With the usage of skills, workspace data and process data, robot programs can be gen- 
erated and kept updated automatically. These key-components are gathered and man- 
aged in a Smart Assembly Model (SAM). SAM is a high- and low-level assembly re- 
source planning system which integrates elements for robotics, humans and processes. 
It also serves as a data collector and sends out commands to robotic devices, machine- 
human interaction systems and other technology in order to control and guide assembly. 

As a data collector, it stores CAD-files of assemblies and parts, 3d-models of assem- 
bly sites, digitized fitting instructions, point clouds, assembly data etc. 

Through its management system it is able to perform a particular set of actions that 
include path planning, Image processing, point cloud processing, digital fitting instruc- 
tion interpretation, animations, digital twin processing etc. 

These days robot programming is a manual task. The programmer needs knowledge 
about the assembly routine and the workspace. Adapting robot programs in a changing 
environment is time consuming and increases non-value-added tasks. 

With skill-based programming an autonomous programming system can be imple- 
mented which reads digitized fitting instructions (FI) and assembly control records 
(ACR) with semantic product data (assembly routine) and overlays instructions with 3d 
factory model (workspace). 

Subsequently, the data is processed with a neural network in order to create a robot 
program consisting of pre-programmed skills and path plans. This automatically gen- 
erated robot program can be reviewed by a robotic specialist and used as a framework 
to do fine-tuning. 
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5 Technology Enhanced Qualification (TEQ) 


Technology enhanced learning enables the worker to learn new processes and work 
steps independently and immersively, without having to be instructed by a supervisor. 
It enables the worker to access work instructions, training and additional information 
on various work steps. The aim is to make the knowledge about the process accessible 
to the employee. The field of Technology Enhanced Learning for Automation serves 
on the one hand as a database for knowledge about process flows and on the other hand 
as a platform for training courses which can be demanded independently by the worker 
with the right technological means (VR glasses, CAVE, tablet, etc.). Corresponding 
learning scenarios are automatically created from digital fitting instructions and simu- 
lated installation paths (called keyframes). Embedded in a virtual shop floor and using 
reduced 3D models derived from the product data, immersive learning units can be 
automatically generated from existing digital fitting instructions. 

After the worker has internalized a process in technology enhanced learning, he can 
train it alone or in groups. The training scenarios are generated from the "Smart As- 
sembly Model". The results of the training can be analyzed and compared with refer- 
ence values. The deviations can be used to determine which assembly steps are mas- 
tered and to what extent (good, sufficient, insufficient). In this way, recommendations 
can be made for training courses. 


6 Smart Human Machine Interaction 


We use a smart system for the interaction between robot and human that employs Aug- 
mented, voice/gesture control etc. Using that system, the operator can interact effi- 
ciently with the robot and its digital twin. Through that interaction data flow in both 
directions, from the user to the robot and vice versa. With the proper processing the 
data is converted into information that both the interacting parts can use. That infor- 
mation processing and exchange leads to gradual continuous development of 
knowledge. Through the abovementioned interaction with the robot and its digital twin 
the operator can see visualized trajectories and next steps, future poses and workspace, 
status, errors etc. Furthermore, collaborative skills like hand over can be controlled us- 
ing the Augmented reality (AR) interface. AR can be used in many fields of everyday 
life and the economy [3]. Reduction of error search and correction can be achieved by 
using AR for assistance tasks in the areas of training [5] [2], assembly, repair [9] or 
automation [13]. AR technologies have great potential in commissioning and mainte- 
nance. For example, when servicing equipment or complex machines, it is important 
for the technician to have all relevant information available. This includes, for example, 
information about what activities are to be performed on this system and what has al- 
ready been done in the past [18]. 

The research on possible applications of AR has been done for many years [11] [16] 
[5] because the AR and AR applications are used in many fields, such as project and 
factory planning. Areas of mechanical engineering and manufacturing automation are 
interesting for AR. Primary AR is used for the visualization of assembly tasks, machine 
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operation, or reparation processes. With a camera (e.g., integrated in the tablet or 
smartphone) the system automatically recognizes the component to which need to be 
repaired and allows all relevant information as well as the exact assembly instructions 
to flow directly into the real image. E.g., the automatic fade-in of the circuit diagram 
while working on electrical system components is conceivable. It is much easier for the 
technician to record very complex system and to check the relevant measured values 
[18]. 

Despite numerous research and development projects in the field of AR technolo- 
gies, it has not yet been widely used in mechanical engineering. The main reason is 
because of the hardware. Since the first experiments from the year 2003 up to the in- 
creasing interest since the year 2008, the hardware available for AR applications was 
too heavy and / or too big or too expensive. 

Now the actual stage of the hardware and software make a developing and a using 
of an AR-application easy and possible for everybody [4]. There is a huge amount of 
free software that can be used easily for complex applications. Thus, it is possible to 
program easy applications for complex mechanical problems [4]. 

The task of check and control can be simplified with a using of the AR as well. A 
practical example of using of the AR for this, is a project for Rolls Royce Germany 
[14]. 


7 Conclusion and Future Work 


The current situation in the maintenance and repair of jet engines is far from optimal. 
The organizational and manager solutions and methods for technical processes, control 
of information and other things are not optimal either. These solutions solve the prob- 
lem ofreliability and safety in the field of engine building and engine maintenance. But 
such decisions require considerable time and material costs. 


This is especially evident with the introduction of new engine models. Thus, classi- 
cal solutions, which imply for example a large number of paper documents, are not 
flexible enough. This is unacceptable in a constantly changing modern world. 

This problem can be formalized as the following points: 


e Changing production environments require flexibility and versatility. 
e Parts, workspaces and assembly routines can change. 
e Adapting to change is time consuming and increases non-value-added tasks. 


So, there is not one, but several similar problems. It cannot be enough to use only 
one new method or approach to solve them all. These problems require a comprehen- 
sive solution. 

This article proposes and discusses the following methods and technological solu- 
tions to overcome the problem: 


1. Object Detection 
2. Collaborative Cognitive Robot (CCR) 
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3. Smart Assembly Model (SAM) 
4. Smart Human Machine Interaction AR 


The article provides a basic description ofthese methods and solutions. Showing their 
main advantages and disadvantages. 

The next step of the authors of the article is the practical implementation of several 
methods in the framework of a small test bench. The main focus will be on the design 
and use of Smart Human Machine Interaction, machine learning and CCR. 
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Abstract. Fiber spraying processes are used in the production of fiber- 
reinforced composites, but are performed mainly manual. Given various 
possibilities for automating such processes, only a few are suitable for 
the production in small and medium-sized enterprises. We present an 
approach on robot-based automation of fiber spraying processes. This 
approach is characterized by its easy programmability of the robot sys- 
tem using the playback programming paradigm. We simulated the reach- 
ability and accessibility of our robot-based automation approach and 
present our prototype system, which can be used for the production of 
small batches of fiber-reinforced composites. 


Keywords: industrial robotics, automation technologies, flexible man- 
ufacturing, small batch production, intuitive robot programming, play- 
back programming, fiber spraying 


1 Introduction 


Fiber spraying processes (FSPs, also called spray lay-up processes) are one of the 
most commonly used production methods for short fiber-reinforced composites 
[1]. While often used in the production of glass and carbon fiber composites, 
FSPs are not commonly used for ceramic matrix composites, such as oxide fiber 
composites (OFC). However, recent research shows that applying an FSP in 
context of manufacturing of OFC can save over one third of the production 
costs [2]. A further reduction of the production costs and an increase of the 
throughput is possible with the automation of such processes. 

Robot-based automation is still only rarely spread in small and medium-sized 
enterprises (SMEs), even though a low cost and high quality production is pos- 
sible with their use [3]. The slight use of robot-based automation is largely due 
to the production of small batches, which is characterized by different phases of 
robot use with frequent reconfiguration and changeover of the robot systems [4]. 
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Fig. 1. Proposed cell for robot-based fiber spraying process. 


A fast reconfiguration can be achieved through a speed up of the robot program- 
ming, which is possible when using intuitive robot programming approaches like 
playback programming [5][6][7]. With the use of such techniques a cost-efficient 
production of small batches could be possible. 

We envision a robot-based automation approach of FSPs, with the aim to 
perform various FSP with different fiber materials in one cell with a shorter 
changeover time. The resulting system should be quickly and spontaneously 
programmed by non-experts in robot programming, which are experts in the 
given process, who we call domain experts. 

In this paper, we present a novel robot-based production system for the 
production of OFC, in which a robot is covering a positive or negative mould 
with a fiber slurry mixture (Figure 1). The system can be extended to other 
kinds of FSP like glass fiber spraying. The robot in the cell is further designed 
to be programmed with the intuitive playback programming paradigm. Section 
2 gives an brief overview of the related work regarding the automation of FSPs 
and their programming frameworks. Then, we explain in Section 3 our robot- 
based automation approach in detail. The simulation and development of our 
prototypical system is described in Section 4. In Section 5, we summarize and 
conclude the paper. 


2 Related Work 


Most of the commercially available fiber spraying systems are manual operated 
systems, on which the various required components are installed, so that they can 


297 


be used like a paint spray gun [8][9]. When used, these manually operated systems 
expose the domain expert to polluted air (overspray, fibers) during the spraying. 
Also, the quality of the produced composite depends heavily on the expertise 
of the operator with every workpiece. So, the quality of produced composites is 
hardly reproducible and the error rate is high. An automation approach should 
put a stop to these disadvantages. 

To the best of our knowledge, there is only one prototypical system for robot- 
based FSPs using off-line programming, which can be characterized by graphical 
programming and virtual reality [10]. Different advantages of this approach are 
given. One of these is, that during the production no worker is exposed to the 
polluted air. Also, the use of a rotary table as an external rotation axis enables 
the processing of complex shapes. Further, layers with uniform thickness can 
be applied repeatedly. However, the off-line programming approach is only of 
limited suitability for small batch production since such programming framework 
requires a special training in robot programming. A new approach should put a 
focus on an intuitive robot programming framework, which exploits the process 
specific knowledge of the operator direct. 

Using on-line programming paradigms could allow a cost-efficient produc- 
tion, for example by hand guiding of the robot tool head, also called kinesthetic 
guiding [6][11]. One of the approaches using kinesthetic guiding is playback pro- 
gramming and has the advantage, that process-specific knowledge of a domain 
expert continues to be incorporated directly into the programming process of the 
robot, as already shown for other processes in [7][11][12][13]. A new approach 
to robot-based automation of FSP for small batch production can also take 
advantages from this programming method. For this reason, a novel approach 
on robot-based automation of FSP should integrate a playback programming 
framework. 


3 Automation of Fiber Spraying 


In the following, we describe our approach in automation of FSPs. First, the 
process itself is explained. Second, we consider important design criteria for an 
automation approach. Then, we explain our system design in detail. Finally, we 
describe how the robot programming framework of the process is designed. 


3.1 Fiber Spraying Process for OFC 


In an FSP for the manufacturing of OFCs aplications, continuous oxidic fiber 
bundles (rovings) are chopped to uniform length by a cutting unit immediately 
before being ejected out of the cutting unit and into a slurry spray, by which 
they are entrained. Due to the angle and the distance between the tool head 
(cutting unit and spray gun) and the mould surface the bundles are infiltrated 
during flight with the slurry, as shown in Figure 2. The infiltrated fiber bundles, 
with slurry being inside the fiber bundles as well as between the fiber bundles 
reach the mould surface and are orientated randomly. 
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Fig. 2. Schematic representation of a manual fiber spraying process. 


If desired, the thus prepared pre-impregnated fiber preform (prepreg) can be 
conditioned in a climate chamber to adjust a defined water content, stacked and 
laminated. In case of flat prepregs, they are stacked to a laminate and laminated 
or directly compressed under pressure, if they are not already corresponding to 
the final form. After consolidation in a drying chamber, the green bodies are 
sintered for two hours at 1225°C. The short fiber bundle reinforcement allows 
more complex shapes than it is the case with fabrics, without fear of damaging 
the fiber bundles or getting delaminations due stresses in the fabrics and a 
resulting resilience. 

As already mentioned this method can be used on the one hand as a primary 
shaping process, such as by spraying on a positive or a negative mold, or as a 
shaping process using the flat prepreg to shape more complex geometries, which 
are not accessible using fabric-based prepregs. On the other hand the spraying 
process build flat prepregs which can be processed like conventional prepregs. 
The big advantage here is the increased drapability of the short fiber prepregs 
compared to the fabric prepregs. Furthermore, the random arrangement of the 
fibres results in area-isotropic mechanical properties. 


3.2 Evaluation and Design Criteria 
The design of an automation approach for an FSP depends on various criteria. 


A very important criterion is the possibility to adjust different parameters of the 
process. One of these parameters is the image of the spay pattern from which the 
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fibers are captured. This spray pattern defines the slurry spray which infiltrates 
the fiber bundles, entrains them towards the mould and so defines also the result- 
ing material properties of the workpiece. An automation approach must be able 
to adjust the slurry spray regarding flow volume and spray pattern, according 
to the objectives of a workpiece. In addition to the slurry spray, the fiber flow 
needs also to be adjustable so that material properties can be modified. This 
plurality of free variables allows the production of a wide variety of workpieces 
with different material properties. 

In addition, economic criteria have to be considered, which are directly re- 
lated to the production costs. One of these criteria is the overall material con- 
sumption of fiber and slurry. Minimizing the material consumption, e.g. the 
amount of overspray which does not form the workpiece, will lead directly to 
lower production costs when using high-expensive fibers. Another criterion is 
the production time, whose minimization will increase the throughput of pro- 
duced components. A further time related criterion is the setup time for the 
production of a single component. This time is required for the configuration of 
the system, when a new component of the same material class has to be pro- 
duced. Reducing the configuration time is essential for the production of small 
batches as described in Section 1. Only if all of the listed criteria are minimized 
to its best, a cost-efficient production of small batches can be reached. 

Finally, process-specific criteria must be considered. The first two criteria are 
to keep a fixed orientation and distance from the tool head to the mould and 
to guarantee the accessibility of the spraying on the mould. This results in an 
optimal application of the fiber slurry mixture. The accessibility is challenging 
due to the fact that we want to accomplish a complete coating even for moulds 
with complex geometries. The last criterion is the ability to generate a homo- 
geneous coverage over the complete mould. Achieving a homogeneous coverage 
leads to uniformly distributed material properties (e.g. the tensile or bending 
strength) over the entire geometry, which can be used as an quality measure for 
a correctly produced workpiece. 


3.3 Robot-based Automation 


As automation approach, we propose a robot system, which manipulates stan- 
dard components of manually operated systems for FSPs, focusing on the pro- 
duction of small components with a maximum size of up to 500 x 500 x 500 mm3. 
First, the structure of the cell is explained, then the concept of the tool head. 
The boundaries of the cell are formed by a cabin of system profiles (W: 
2000 mm x H: 2200 mm x D: 1800 mm). The cabin is sealed by windows and 
resealable doors made of acrylic glass, so that overspray and fibers are contained 
within the cell and the production can be monitored from outside. In addition, all 
control elements are mounted outside the cell so that the presence of a person 
inside the cell is not necessary while the FSP is running. Within the cell, an 
underfloor downdraft table (W: 1080 mm x H: 460 mm x D: 1195 mm) is set 
up, where a mould can be positioned. In the middle of the downdraft table a 
programmable turntable is installed allowing the rotation of the mould through 
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Fig. 3. Proposed concept of a robot-based fiber spraying cell. Left: Interior of the fiber 
spraying cell. Right: Tool head design with possibility of kinesthetic guiding. 


this external axis. Additional degrees of freedom can be added by extending 
the turntable to a two-axis turntable. The downdraft table limits the intended 
working space of a six degree of freedom robot, which is mounted in front of the 
downdraft table (Figure 3, left). 


The possibility of programming by kinesthetic guiding was taken into con- 
sideration in the design of the tool head (Figure 3, right). A force/torque sensor 
is mounted in z-direction from the flange. A handle is additionally attached on 
the force/torque sensor, through which the guiding of the robot arm is enabled. 
The automatic spray gun is installed in front of the tool head. The slurry flow 
volume as well as the spray pattern are adjusted by a programmable system of 
proportional pressure regulators. The fibers are fed via a cutter mounted over 
the spray gun. The fiber roving supply is mounted on the ceiling of the cell. The 
amount of fiber throughput is also controlled by the proportional pressure regu- 
lators. A diffractive optical element projects a pattern onto the mould allowing 
the estimation of the spray cone and thus the application of material without a 
material flow. The resulting tool head has a weight of 4 kg, which can also be 
mounted on lightweight robots. 


Complying a fixed orientation and distance to the mould and achieving ho- 
mogeneous layer thicknesses is possible due to the programmability and repeata- 
bility of robot movements. At the same time an increase in the throughput of 
produced ceramics can be achieved by the autonomous execution of robot pro- 
grams. The accessibility of the mould is guaranteed by the use of a turntable, 
which also reduces the robot wear. 
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3.4 Robot Programming 


An approach in robot-based automation for small batches requires a program- 
ming technique, which can quickly provide robot programs for covering various 
moulds fast. Textually created robot programs require the specialized knowledge 
of a robotics expert. Further, a new robot program for each workpiece or groups 
of workpieces in case of simple geometries (plates, tubes) is needed. For this rea- 
son, the robot has been equipped with the ability to be guided kinesthetically 
by a domain expert. The kinesthetically guiding allows the use of the playback 
programming paradigm [6]. Given sensor data from a force/torque sensor, the 
robot moves in answer to this input of the operator, who can be a domain ex- 
pert. While the robot arm is guided by the operator through the spraying task, 
the system records robot configurations in a fixed cycle and generates a robot 
program with the demonstrated trajectory. After this programming stage, the 
robot program can be played back so that he can execute the intended task. 

The use of the playback programming paradigm allows the quick teaching 
of trajectories for geometrically complex moulds, which would be time and cost 
consuming by a textual programming approach. Also, kinesthetic programming 
is intuitive and can be done by domain experts. Further, our system allows, that 
the playback programming can be done without material flow and is so resource- 
saving. The resulting program can be used for the automatic production while 
the domain expert can address oneself to other tasks. Finally the reproducibility 
of robot movements guarantees that a fault-free program can be executed again 
and thus reduces the error rate of the production. 


4 Prototypical System Development 


In this section, we describe the development of our prototypical system. First, 
we describe the analysis of our approach through simulation. Then, we show how 
our first prototypical system was designed and give an overview about its use. 


4.1 Simulation 


Given geometrical data of the cell and tool head in form of CAD-models, we 
simulate the reachability of the robot for different poses. Our aim is to check, 
whether the optimal distance for covering a mould is achievable. The optimal 
distance is given by our first application, the process given in [2]. With the opti- 
mal distance, we generate distributed poses on a hemisphere around the position 
where the mould will be placed. After that, we check if the robot kinematic can 
reach the poses on the hemisphere. Additionally, we let the simulated robot move 
into the reachable positions, using a collision check between the robot and the 
downdraft table to guarantee the accessibility of the poses. 

For simulation we use the Stdéubli Robotic Suite [14] (Figure 4, left). We 
sample 500 poses and a distance from spray jet to mould of 400 mm is used, 
resulting in a overall distance from flange to the hemisphere center point of 645 
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Fig. 4. Simulation of our approach on robot-based fiber spraying process. Left: Simu- 
lation setup with reachable poses marked as black dots. Right: Detailed results of our 
simulation with classification into not reachable poses (red), reachable poses but the 
movement causes a collision (blue) and collision-free reachable poses (green). 


mm. 263 of the generated poses were reachable, in which 85 result in collisions 
with the robot cell while moving. So, the remaining 178 poses are accessible 
from the robot without collision. Eight collision-related poses are characterized 
by being in an area, which is dominated by accessible poses. The results are 
shown in Figure 4, right. 

One of our conclusions from the simulation is, that only with the use of 
a turntable geometrical high complex moulds with the required maximum size 
can be covered complete. Another conclusion is, that for plain geometries, the 
reachability and accessibility for the optimal distance can be guaranteed with 
our concrete approach. Our approach, which uses a combination of turntable 
and robot, is thus sufficient for the required process and workpieces. 


4.2 Prototypical System 


For our first prototypical system we use a Stäubli TX2-90L [15] as robot and an 
ISEL IT 116 [16] controller for the turntable. The control of the different pres- 
sures needed for the material feed is given by a system of proportional pressure 
regulators. Slurry output is enabled through a spray gun, which is a commer- 
cially available system suited for the use mounted on a robot. The turntable and 
pressure regulators can also be controlled by the manual robot control pendant. 
Five pressures can be configured via the regulators: two defining the slurry, one 
for setting the actual slurry output to ON/OFF, one for controlling the volume 
flow of the slurry, and the last for the rotation speed of the chopper unit. A 
force/torque sensor provides the input data for the playback programming. All 
components are connected and controlled via the controller of the robot. 
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Fig. 5. An example use of our prototypical system for robot-based fiber spraying pro- 
cess. The operator utilizes the manual control pendant for programming. 


Given a mould to cover, the operator teaches the required trajectory for 
the robot by kinesthetic guiding. The playback programing without material 
feed is defined through the user interface of the manual pendant control before 
the programming stage begins. Also, the turntable and the pressure regulators 
are configurable with the manual pendant control, as shown in Figure 5. After 
teaching, the trajectory can be stored and reloaded for a later production. In 
the execution stage, the trajectory is replayed with temporal synchronization 
between the robot, the regulators and the turntable. 

This prototypical system allows the playback programming of geometrical 
complex trajectories with a one shot demonstration, resulting in the fast pro- 
duction of small batches of new workpieces. The quality of the programmed 
trajectory depends on the skill of the system operator and is suitable for work- 
ers with expertise in the given process, but can easily used by them without 
special training in robot programming. 


5 Conclusion and Future Work 


In this paper, a novel robot-based automation approach for production of fiber 
composites with FSPs is proposed. The developed prototypical system can be 
used for the production of OFCs. Reachability and accessibility is guaranteed 
through simulation. The presented playback programming framework allows the 
programming of the robot in a new way by a domain expert without special 
training in robot programming. However, the presented playback programming 
framework is still not sufficient for a small batch production of geometrical high 
complex workpieces. For this reason, we will use in future work a playback pro- 
gramming framework which allows the intuitive programming of such tasks using 
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video editing concepts [5]. Also, this approach will be extended with functional- 
ities for supporting the domain expert, like the simulation of the process or the 
optimization of the programmed trajectory before execution. 
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Abstract. Speech-based robot instruction is a promising field in pri- 
vate households and in small and medium-sized enterprises, because it 
facilitates a comfortable way of communicating with robot systems, even 
while the users’ hands are occupied. An essential problem in transforming 
the speech-based instructions into complex robot motions is the valida- 
tion of the resulting motions regarding feasibility. This emerges from the 
fact that the user may not be fully aware of the capabilities of the robot 
including its gripper in a given working environment. We present an ap- 
proach that tackles this problem by utilizing dynamic affordances for the 
disambiguation and elemental validation of speech-based instructions. 


Keywords: Situated Dialogue, Affordances, Disambiguation 


1 Introduction 


One long term goal in current robotic research is the development of robot 
systems which have approximately the same cognitive, communicational, and 
handling abilities as humans. As part of this ongoing development, application 
domains for robot systems shall be expanded, from industrial settings with sepa- 
rated working cells, fixed object positions, and preprogrammed motions towards 
a flexible usage in small or medium-sized enterprises or private households. 

An intuitive way of communicating with such a robot system is to instruct 
it via spoken commands. Advantages of this approach are that it facilitates the 
instruction of a robot without having to learn a robot programming language 
and an instruction while the users’ hands are occupied. In user studies we noticed 
that ambiguities are a problem that occurs while instructing a robot and that 
in most cases ambiguities result from differences in the mental representation 
of the user and the robot system. For example, users omit parameters in an 
instruction, which seem obvious from their perspective. Informing the user about 
missing or incorrect parameters can only be performed well if a common ground 
is present. One way of doing this is by utilizing affordances which can be seen as 
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a functionality of objects, e.g. graspability or liftability. In the case of force based 
object manipulation this is a helpful tool, because affordances of the workpiece, 
the robot arm, and the tool in use can be evaluated for a given instruction. 

The main contribution of this paper is a concept that is able to disambiguate 
and validate instructions based on dynamic affordances regarding the robot, its 
tool, and the workpiece. The scientific gain of this approach is bipartite: First, 
affordances are used in a preprocessing step to reduce the parameter space and 
even empowers a system to execute motions emerging from incomplete com- 
mands. Second, by using dynamic affordances the feasability of instructions can 
be applied in changing working environments and for various hardware. All this is 
done to investigate the application of affordances for previously unknown tasks, 
instead of relying on plans or pre-trained models. Moreover, utilizing affordances 
facilitates a communication with the user on a common ground, which is helpful 
in the case of sensor-based robot motions. 

In Section 2, an affordance overview is given, highlighting approaches regard- 
ing affordances, disambiguation, and a combination of these. In Section 3, the 
approach is described in detail. Section 4 contains the description and results 
from our user study verifying the validity and usefulness of our approach and 
in Section 5 a summary of the approach is given, as well as a glance at future 
work. 


2 Related Work 


In this chapter, approaches are presented that deal with the definition and use 
of affordances, disambiguation of spoken instructions in human-robot dialogue, 
and concepts combining these areas. We also describe in what way our approach 
differs from the state of the art. 

Experts and non-experts may not always be aware of the capabilities of a 
robot system while instructing it. This holds true especially in force-based tasks, 
like peg-in-hole or cutting tasks, where force parameters have to be considered as 
well. A well studied approach for defining and evaluating these functionalities are 
affordances introduced by [6], defining what can or cannot be done with an ob- 
ject [20]. The concept of affordances has been part of the scientific community for 
several decades. It leads to a lot of publications surveyed regarding psychology, 
neuroscience and robotics [11], human-robot interaction [21], and developmental 
robotics [19]. Being dissatisfied from the variety of formalizations, a computa- 
tional affordance model is given in [27] which classifies approaches regarding 
perception (selective attention, temporality, order, level, perspective), structure 
(chaining, competitive, abstraction), and development (exploitation, generaliza- 
tion, prediction, learning, acquisition). Affordance learning has also been studied 
for a lot of applications, for example grasping [22], [5], traversability [2], [12], 
[24], or tool-use [1], [28]. 

Disambiguation has also been discussed a lot along the last decades [15], 
[14], [17], [25]. Most approaches try to solve ambiguities by using clarification 
questions based on yes/no questions [4], [9], listing all options [16], or generic 
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Fig. 1. Concept overview: Blue arrows highlight the main information flow and grey 
arrows the control flow between the logics and the hardware/data base . 


WH-questions [23]. In search for a general approach for disambiguation, [25] 
suggests three design recommendations. Robots should present all options only if 
the number of options is less than three, otherwise robot systems should generate 
yes/no or WH-questions (D1). Clarification requests should be formulated in a 
way, that the user thinks the robot understood the goal of the instruction (D2). 
Robots should use pragmatically appropriate phrasings (D3). 

Investigating the combination of affordances and disambiguation in the field 
of speech-based interaction has been poorly investigated by just a few approaches 
[20], [8], [10], [3]. In [10] the application of affordances for co-reference resolution 
in speech based commands was mentioned, but not further investigated. In [8] 
ambiguous commands where solved utilizing the affordance concept together 
with a prediction based on previous commands. In [3] a neural network learned 
with a huge text corpora was used to disambiguate incomplete instructions. 

In difference to the introduced approaches, we present a system which ap- 
plies the concept of affordances in form of pre-conditions [13] for disambiguating 
and validating spoken instructions without relying on previous recorded data or 
text corpora, but solely on dynamic affordances, which means that their state 
can change over time or depending on the hardware set-up. For example, the 
reachability of an object can change during a manipulation task (hidden behind 
other work pieces) or not reachable by the robot arm currently installed. 


3 System Overview 


This section describes the main aspects of our framework. After giving a short 
framework overview (Section 3.1), we describe the modules in more detail, which 
are part of the main information flow. Besides explaining the dialogue module 
(Section 3.2), we present our disambiguation approach implemented in the iden- 
tification and validation module (Section 3.3 and Section 3.4), and explain how 
and what kind of feedback is generated by the system (Section 3.5). 
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3.1 Framework Overview 


Our framework for SPeech based Instruction of RObot systems (SPIRO) is 
divided in static modules: dialogue-manager (DM), identification module, vali- 
dation module, feedback module, and changeable modules: The active robot (R), 
the attached tool (T), and the knowledge data base (DB) (see Figure 1). 

The information flow starts by a spoken user instruction which is recognized 
and semantically enriched by the DM. This information is then passed to the 
identification module, which has the purpose to identify the motion contained 
in the instruction along with the necessary parameters based on attached affor- 
dances. The validation module then checks the given input regarding the affor- 
dances in the current world state. The validated information is then passed to 
the feedback module which passes the appropriate motion data to the hardware 
and the acknowledgement to the DM. If an ambiguous or, in terms of the do- 
main, incorrect instruction is recognized by the DM, identification or validation 
module, the problematic affordance is tagged as non valid or ambiguous and an 
appropriate clarification question is generated by the DM. In this case, no robot 
motion is generated. 

To allow a flexible use of our system we designed it in a way, that the robot, 
tool, and database can be exchanged easily. Being aware that switching the robot 
arm does not happen as often as switching the tool or underlying database we 
nevertheless included it to be also able to cover even rare cases. 


3.2 Dialogue Manager 


The DM is implemented using the ontology based dialogue platform Semvoz', 


because it offers a fast design and adjustment of dialogue system, combined 
with the Nuance? framework, because of its valuable speech-to-text and speech- 
recognition features. The DM has four purposes: Automatic speech recognition 
(ASR), semantic role labeling (SRL) [18] of incoming user instructions based on 
pre-defined ontologies, i.e. labeling red as a colour or shove as a verb, generating 
verbal phrases or starting a dialogue based on data provided by the feedback 
module, and returning system answers in an acoustic way (TTS). Here, an on- 
tology is an ”is a”-hierarchy which allows the user to add semantic knowledge 
to the spoken instructions. 

The ASR module converts the spoken user instruction into word set hypoth- 
esis. The hypothesis with the highest probability is then provided to the system 
for further processing. For example, an instruction like: ” Place the cup on the 
plate!” is transformed into an instruction tuple 


I = (verb( Place), workpiece(cup), tool(gripper), destination(plate)). 


This SRL is done automatically by Semvox as long as semantic roles like verb, 
tool, etc. are pre defined. The generated instruction tuple is then sent to the 


1 https: //www-.semvox.de/en/technologies/odp-s3-the-basis-for-interactive-systems / 
? https: //www.semvox.de/en/technologies /nuance-speech-recognition-tts / 
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identification module and the validation module to compute the appropriate 
answer to the instruction. 

The other purpose is to generate verbal feedback for the user in one of the 
four cases: Each parameter of the instruction was specified correctly (C1), the 
system estimated a disambiguation of an incomplete or ambiguous instruction 
(C2), the system was not able to disambiguate parameters of the instruction 
(C3), the system recognized incorrect instruction parameters (C4). 

In case C1 an acknowledgement like ” Yes!” or ” Yes, of course!” is sent to 
the user. In case C2 the disambiguated instruction is sent to the user with 
the opportunity to interrupt the system by saying ”No!” or ”Stop!” and the 
motion is executed by the robot. In case C3, a clarification question based on 
the ambiguous parameters is generated and in case C4 the user is informed about 
the false parameters. 


3.3 Identification Module 


This module has the purpose to identify the motion, i.e. finding an appropriate 
motion template in the DB, and accompanied parameters intended by the user 
instruction. Given the instruction tupel Z recognized by the DM, the suitable 
motion template is searched and filled using the affordance filtering approach [7] 
performed by the two sub modules: Motion identification (T1) and parameter 
identification (T2) (see Figure 2). 

In T1 the intended motion has to be identified. Here, two cases arise. Case 
1: The verb is mentioned. In this case, the system directly picks the appropriate 
motion templates for this verb from the DB and continues with T2. Case 2: No 
verb is mentioned. In this case, we consider two possibilities: The user implicitly 
uses the verb from the previous instruction or he explicitly did not mention the 
verb because its obvious for him. The implicit case can easily be solved with 
Semvox, which allows an auto-filling of the verb slot if the current instruction 
has the same structure as the previous instruction, which then automatically 
leads us to Case 1. The explicit case leads to a work around. The system then 
has a look at the other specified parameters and tries to find a verb which 
maps to these parameters, both syntactically and affordance-based. If this still 
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does not lead to a single verb candidate, the verb slot is filled with the possible 
candidates, tagged as ambiguous and the augmented instruction tupel is send to 
the validation module. Imagine the user just saying: ” The big screw!”. Instead 
of not answering at all or responding: ” What do you want me to do with it?”, 
we have a look at the valid affordance at this time step and try to disambiguate 
the opportunities for the verb slot. If this is not possible, we tag the verb slot as 
ambiguous. 

After the motion is identified, the remaining parameters are filled using the 
motion template provided by the DB. In the simple case all the parameters are 
defined and we send the augmented instruction tupel to the validation module. 
If one of the parameters is ambiguous, a disambiguation is necessary. Here again, 
we are using affordances to filter ambiguous or even generate missing parameters. 
Imagine the instruction: ” Shove to the left!” Normally, we would have to ask the 
user which object he wants us to shove to the left, but maybe there is only one 
object in our workspace that is shove able. If there is more than one option left, 
we tag the affected parameter slots as ambiguous before sending the augmented 
instruction tupel to the validation module. 


3.4 Validation Module 


The purpose of the validation module is to evaluate the instruction parameters 
by evaluating the affordances for the given world state and tag them with the 
description: VALID, INVALID, AMBIGUOUS, and DISAMBIGUATED. Using dynamic 
affordances is necessary, because in some cases valid defined instructions may not 
be executable, because the object is out of reach for the robot arm or the gripper 
is not suitable for grasping the workpiece. A parameter is tagged as VALID, if it 
is distinct regarding the necessary affordance and has the correct type regarding 
the verb frame. A parameter is tagged INVALID, if it does not have the correct 
type regarding the verb frame. A parameter is tagged AMBIGUOUS if a set of 
possible real world objects is possible for it and they are of the right type. A 
parameter is tagged DISAMBIGUATED if the identification and validation module 
managed to disambiguate the user instruction. The tagged instructions are then 
send to the feedback module for further investigations. 


3.5 Feedback Module 


The feedback module is responsible for transforming validated instructions into 
either a text-based response for the DM or a text-based response for the DM 
in combination with robot motion parameters for the motion generator. What 
kind of feedback is given depends on the validity of the parameters slots of the 
instruction. In the case that the validity slot of all parameters holds VALID the 
feedback is given in form of an acknowledgement phrase (”Ok!”, ”Sure!”, ” Yes!” ) 
send to the DM together with motion parameters sent to the motion generator, 
which generates a hybrid motion based the approach presented in [26]. If the 
validity slots hold DISAMBIGUATED as well as VALID, the instructions are sent to 
the DM for a spoken response before the motion is executed. If the validity slot 


Fig. 3. 1: Example working environment for instructing a robot arm. 2 - 4: Intermediate 
state of the key holder. 5: Finalized key holder. 


of at least one parameter of an instruction holds I, the user is told to correct 
the specific parameter and no robot motion is generated. If one of the given 
parameter holds AMBIGUOUS, a clarification question based on D1 - D3, defined 
in section 2, is generated and send to the DM. This process is repeated for one 
property at a time until the presented instruction is valid regarding all occuring 
parameters. 


4 User Study 


In this section, a user study is presented which was conducted to gain information 
on how users react to incomplete and ambiguous instructions and why they 
choose their reaction. After a presentation of the user study setup (Section 4.1), 
the results are evaluated as well as discussed (Section 4.2). 


4.1 Setup 


The study was set up using the online tool Survey Monkey? and was divided in 
two main parts. 

In the first part, we wanted to gain information about the necessity of a 
validation component and presented the participants the working environment 
shown in Figure 3, 1, together with the request to instruct the shown robot to 
place a motor with a given weight of 4 kg from location P1 to location P2. This 
was asked to see, if users are aware of the capabilities of lightweight robot arms 
and at the same time if a validation or correction of user instructions is necessary 
in cases where a robot is not able to perform an action. 

In the second part, the participants were put in the position of an assistant 
robot, which helps its co-worker to build a key holder (see Figure 3, 2 - 5). The 
tasks for the participants where chosen in that manner, because they are either 
better executable by a robot due its precision or because the user typically does 


3 https: //www.surveymonkey.de/ 
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Table 1. Votes for the different reaction types (RT1 - RT6) for the tasks marking, 
inserting, and cleaning in the second part of the user study. 


task RT1 RT2 RT3 RT4 RT5S RT6 
marking 2 3 1 7 3 0 
inserting 3 0 1 9 3 0 
cleaning 2 1 1 4 7 


not want to perform the task. For each instructed task, the participants had an 
image of the current state of the key holder, an instruction of his co-worker, and 
possible tools or objects to perform the tasks. They were then asked to choose 
between a set of reaction types: spoken disambiguation of three candidates and 
no motion generation (RT1), spoken disambiguation of two candidates and no 
motion execution (RT2), spoken query for the missing/ambiguous parameter 
and no motion execution (RT3), spoken feedback in form of the disambiguated 
instruction and motion execution (RT4), Spoken acknowledgement and motion 
execution (RT5), no spoken feedback and motion execution (RT6). 

The first instruction is: ”Mark the board with pattern M1!”, the informa- 
tion what M1 looks like is given as well as the tools: pencil, text marker and 
an permanent marker. The participants are then told that their co-worker in- 
serted holes at the marked spots and that the new instruction is: ”Insert the 
hooks into the holes!”, followed by a set of hooks with different lengths. After 
that the participants are told that their co-worker screwed in the hooks, took 
the key holder away, leaving a dirty workplace with the comment: ” Clean the 
workplace!” . They are also given a set of tools suitable for cleaning a workplace: 
sponge, cloth and hand brush. 


4.2 Results 


Overall 15 participants joined the first part of the study. Although experts as 
well as non-experts where among the participants, all of them instructed the 
robot to move the motor from place P1 to place P2 without considering if the 
robot arm is even capable of doing it. This result clearly shows, that a validation 
even in simple tasks, like moving an object, is necessary. Without a validation, 
this would normally lead to an incorrect execution of the motion resulting in 
frustration of the user or even damage. Using our affordance based validation, 
we are not only able to perceive this error, but also to formulate a spoken solution 
to this problem, in this case: ”I can not lift the motor since it is too heavy!”. 
Overall 16 participants joined the second part of the study. The results are 
shown in Table 1. In the marking task most of the participants (43%) favoured 
RT4, especially the use of the pencil. One participant even wrote, that mention- 
ing the instruction he understood, enables his co-worker to intervene, if he is not 
pleased with the disambiguation. In the inserting task, even more (56%) partici- 
pants chose the reaction RT4. Here, two participants justified their reactions by 
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saying that the user is able to interrupt the motion, if he is no pleased with it. 
The remaining participants also mentioned that they made their decision based 
on the physical properties of the hooks and the board, thus, in a way our system 
would work as well. In the cleaning task, the majority of the participant chose 
RT5 as their favourite reaction and RT4 only got the second place. Based on 
the participants justifications, the brush is in this case the best tool, or at least 
as good as the cloth, but no participant mentioned, that either the sponge or 
the cloth would be the best tool. This leads us to the conclusion that a listing 
of opportunities is only useful, if the evaluation of affordances leads to a similar 
rating. 


5 Conclusion 


We present a concept for an affordance based disambiguation and validation 
of incomplete, incorrect, and ambiguous spoken robot instructions. By utilizing 
dynamic affordances the system is able to reduce the amount of candidates in 
the case of ambiguous instructions or even solve the ambiguity. Moreover, using 
dynamic affordances allows not only validating semantically correct instructions, 
but also contains the information of the problems’ source on a human under- 
standable level. Limits of our approach are the common problem of not being 
able to reduce the solution space and the complexity of computing the dynamic 
affordances, while the latter one is still part of current research. 

To evaluate our concept, we also conducted a user study where we collected 
information on how and why users react to incomplete and ambiguous reaction, 
and if they are aware of robot capabilities even in simple tasks. On the one hand, 
the results correspond to the presented concept. On the other hand, we also found 
a way to improve our concept by using acknowledgements if the disambiguated 
parameters are rated high enough. 

In the future, we will implement the whole concept and perform further user 
studies to be able to improve our dialogue system. 
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Abstract. Positioning objects in industrial handling applications is of- 
ten compromised by elasticity-induced oscillations reducing the possible 
motion time and thereby the performance and profitability of the au- 
tomation solution. Existing approaches for oscillation reduction mostly 
focus on the elasticity of the handling system itself, i.e. the robot struc- 
ture. Depending on the task, elastic parts or elastic grippers like suction 
cups strongly influence the oscillation and prevent faster positioning. In 
this paper, the problem is investigated exemplarily with a typical han- 
dling robot and an additional end effector setup representing the elastic 
load. The handling object is modeled as a base-excited spring and mass, 
making the proposed approach independent from the robot structure. 
A model-based feed-forward control based on differential flatness and a 
machine-learning method are used to reduce oscillations solely with a 
modification of the end effector trajectory of the robot. Both methods 
achieve a reduction of oscillation amplitudes of 85% for the test setup, 
promising a significant increase in performance. Further investigations on 
the uncertainty of the parameterization prove the applicability of the not 
yet widely-used learning approach in the field of oscillation reduction. 


Keywords: flatness-based control, reinforcement learning, double deep 
Q-network, trajectory optimization, oscillation reduction 


1 Introduction 


Pick and place processes are one of the most common applications in automation 
technology and robotics and their extensive use in modern production enables 
significant benefits even by the smallest improvements in accuracy or speed of 
motion. For this reason, various efforts are being made to increase the perfor- 
mance of industrial robots and, above all, to optimize the accuracy of their end 
effector positioning. 

Classical methods to achieve this are e.g. model-based feedforward control 
of actuator speed and torque for a given motion profile. The motion profile 
can further be optimized regarding motion time and smoothness based on the 
robot dynamics, as shown in [13] for high-speed pick and place applications. 


© The Editor(s) (if applicable) and The Author(s) 2020 
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Handling and Industrial Robotics, https://doi.org/10.1007/978-3-662-61755-7_29 
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A generalization of feedforward control is given by flatness-based control which 
theoretically makes it possible to design a motion profile for complete vibration 
reduction [6]. The implementation for high-speed positioning is e.g. presented in 
[1] for a linear flexible motion system using trapezoidal motion profiles. The con- 
cept of adaptive input shaping from [14] allows a more robust parameterization 
than the feedforward control by using frequency parameters instead of specific 
physical parameters, which can be difficult to obtain. These approaches all share 
the necessity of a correct robot model and an identification of the required model 
parameters, making them partially impractical in their implementation. 

The methods of machine learning on the contrary do not require system 
knowledge or any kind of parameter identification, since they are purely based 
on recorded data. An overview of different learning methods for the feedforward 
control of a 7-DoF (degrees of freedom) serial link robot arm and an assessment 
of their performance is given in [9]. 

The presented methods until here all have the common goal of increasing the 
accuracy of the robot and its end effector. The objects to be manipulated, espe- 
cially without rigid coupling, are not part of the considerations. For this reason, 
only the oscillations occurring at the robot structure can be reduced. In [2] the 
reduction of shear forces between a suction pad gripper and the object is shown 
by adjusting the orientation of the end effector while moving. The underlying 
assumption of the availability of degrees of freedom in the orientation is however 
not always complied with by the robot or the application. Similiar to the pursued 
approach in this paper, [3] proofs that the oscillations of a rope-coupled mass 
attached to a flying drone can be reduced by methods of reinforcement learning. 

To extend these results to the oscillation reduction in the positioning of 
elastically coupled objects with robot manipulators, the contributions of this 
paper are 

1. a structure-independent flatness-based control for handling elastic objects, 
2. the comparison of this method with state-of-the-art machine learning. 
The remainder of this paper is organized as follows: A brief overview of the 
theoretical and mathematical background of the presented methods is given in 
Sec. 2 and their implementation for the specific system at hand is given in Sec. 3. 
The experimental setup and results are presented in Sec. 4, demonstrating the 
high potential of the proposed methods. Sec. 5 concludes the paper. 


2 Background 


First, a brief summary of flatness-based control, reinforcement learning in general 
and Double Deep Q-Network (Double DQN), the algorithm used within this 
work, is given. 


2.1 Flatness-Based Control 


Dynamic systems with the property of differential flatness are characterized by 
the existence of a flat output yf which in the single input single output case 
allows transformations 
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w=, (uti) and usw (ui) O 


leading to a representation where the state vector x and the input u can be 
expressed by means of a flat output yp and a finite number of its derivatives up 
to the order of the the system’s relative degree k [6]. While no general method for 
finding a flat output of a nonlinear dynamic system exists, in the case of linear 
controllable systems the controllable canonical form allows determining the flat 
output directly [6]. After a flat output is found, one can use (1) to calculate an 
input signal u(t) that leads to the desired output trajectory y(t) for the system. 


2.2 Reinforcement Learning 


In the context of reinforcement learning an agent finds itself in a state S, and 
interacts with its environment through an action A; derived by a policy 7. Acting 
on its environment results in a change from state S+ to St+ı and an observation 
of a reward R;;ı, valuing the chosen action and new state. The agent’s objective 
is to maximize the discounted return 


Gt = Rigi +7Rego + PRs t+. = Soy Repke. (2) 
k=0 


where y € [0, 1) describes the importance of future rewards [11]. The true value 
for a specific state s and action a, under a policy 7 then becomes 


S:= 8, A4 =a ’ (3) 


Qr(s,a) = Er [Gi St = s, A: = a] = Er È YF Ritk+1 


k=0 


where the optimal value is denoted as Q, (s,a) = max, Qr(s,a) and the optimal 
policy is made up of choosing always the highest valued action in each state [12]. 
In case of large or continuous state spaces, it becomes inadequate to store the 
Q-value for each discrete time step t separately. Instead, the objective becomes 
learning a parameterized value function Q(s,a;0), were O denotes the parame- 
terization. The function approximator can be linear in its weights 0 or nonlinear 
(e.g. an artificial neural network, ANN) [11]. The proposed approach uses an 
algorithm from the latter class of methods. 
The standard update rule for the parameterized Q-learning algorithm is 


0:41 =O + a [Ur — Q(S:, At; 9:)] Vo, Q(St, At; 0+), (4) 
where a is the step size and the target U; is defined by 


Ur = Regi + ymax Q(St41, a; 0+). (5) 


In the Double DQN approach from [12], two ANNs are used for function ap- 
proximation. Both networks share the same network architecture but deviate in 
their weight vectors, resulting in the target 


UpoubleDQn,t = Rırı + YQ(St41, arg max Q(Si41, 4; 0t); OF ). (6) 
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Here, 0; parameterizes the online network and 6, the target network. Inspired 
by Double Q-Learning [4] the online network is used to select the current ac- 
tion, while the target network estimates its value. Similar to [8] the weights of 
the target network 0; are updated by or copied from the online network 4 
every N time steps and are kept constant otherwise. To break correlations be- 
tween training samples it is beneficial to use experience replay [5], where tuples 
of observed transitions (S;, Ar, S:41, Rt+1, terminal) are stored in a buffer with 
size Ng and sampled uniformly in order to train the network; where ‘terminal’ 
describes whether $;41 is a terminal state. 


3 Implementation 


As motivated in Sec. 1, the considered scenario is a pick and place task where the 
object is not rigidly but elastically coupled to the tool center point (TCP), e.g. 
by elastic grippers like suction cups. Therefore, the coupling between the robot’s 
TCP and the object is modeled as a base-excited mass, illustrated in Fig. 1 (a), 
where the robot structure is assumed to be rigid. To investigate the principal 
relations, a 1-DoF replacement model is considered for the robot and the TCP. 

Thereby y and y denote the objects Cartesian position and velocity, while u 


(b) 


Fig. 1. (a) Base-excited mass and (b) testbed construction. 


and ü are the TCP position and velocity. Furthermore, c and d denote spring 
stiffness and damping, which altogether results in the differential equation 
my + dý + cy = dù + cu, (7) 


with disappearing initial values for y and u and their derivatives. This model is 
employed for both the flatness-based control and the training of the agent. 


3.1 Flatness-Based Control 


Equation (7) describes a linear system with a relative degree of one. A transfor- 
mation into its corresponding state space representation leads to a state space 
model in controllable canonical form (with yr Æ y) 


i) = Ce-a) GTO. ©) 


v= (6) (4). 0 


Yf 
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where yr represents a flat output of the system (7). The dependency on time t 
for states yp and Yr, input u and output y is omitted for a better readability. For 
the objective of a rest-to-rest motion two approaches are pursued: 


Approach 1 (Neglection of damping): Neglecting d in (7) results to 
my + cy = cu, (10) 


where the physical state y itself already is a flat output and the control input u 
can be obtained after dividing both sides by c. 


Approach 2: Instead of transforming the whole reference trajectory Yrer for 
the physical output y into flat coordinates, only the initial and end conditions 
for the output are transformed. Since the goal is to achieve a rest-to-rest motion 
for the output y, only the dynamics at start and end of the motion have to be 
constraint. After transforming initial and end values of the physical trajectory 
into flat coordinates, a polynomial of degree 9 for the reference trajectory yf ref 
in flat coordinates is used. The constraints on derivatives up to order 4 avoid 
exciting unmodeled robot dynamics. For the transformation of initial as well as 
end conditions (9) is used to obtain 


Y(T) =yr = Sye(T) + FHL) and HT) =9r = ýT) + $ü(T) (11) 


with T denoting the end of motion. In condition (11) the second derivative of yf 
appears, which then can be substituted by the last row in (8) 


yr = £T) + 4 (—£ye(T) — Lie(T) + u(T)). (12) 


At the end of motion the mass has to be in rest at a specific position, which 
leads to the conditions 


yT)=yr and yT)=0 (13) 


and, in addition, the robot’s TCP should simultaneously arrive at the same end 
position as the mass, leading to 


u(T) = y(T) = yr. (14) 
Finally, the transformation for position and velocity at time T appears to be 
y(T)= Fyr and y(T') =0 (15) 


while initial values equal zero. 


3.2 Double Deep Q-Network 


For the learning algorithm a Double Deep Q-Network is used where the agent is 
trained in a simulated environment using the same mathematical model of (7) 
as for the flatness-based control approach. The agent’s objective is to learn a 
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correction for the reference trajectory Yrer, which is again a polynomial of degree 
9, but in physical coordinates, leading to the position profile 


p(t) = Yrer (t) + a(s), (16) 


where a(s) denotes the agent’s action in state s. The agent selects out of 101 
equally spaced positions a ranging from —0.03 m to 0.03 m. For the state repre- 
sentation the error in position and velocity between the mass and the reference 


trajectory Bu a 
e= te 3 = (‘) (17) 
Yref — Y e 


is used, since learning can be sped up if the ANNs inputs have a mean of zero 
[10]. Further, the reward signal penalizes error in distance, velocity and large 
changes by the agent through 


r=-{e} - {e} - 0.5{a}, (18) 


where SI-units are removed with the {-} operator. The network consists of three 
fully connected layers, two inputs, 48 neurons in the hidden layer and 101 neurons 
in the output layer. Training in a simulated environment led to a discontinuous 
position profile, which was unsuitable for highly dynamic motions. Hence, a 
polynomial of degree 19 was fitted to the position profile p(t) altered by the 
agent. Finally, Table 1 summarizes the most important hyper parameters. 


Table 1. Hyper parameters for the Double Deep Q-Network. 


Parameter Symbol|Occurrence|Value 
Learning rate a (4) 1077 
Discount factor y (2)(3)(5)(6) | 0.99 
Size of replay buffer Ng - 10° 
Size of mini batch - - 128 
Update period of target network N - 30 
Exploration decay € - 0.0036 


4 Experiments 


The effectiveness of both presented methodologies is studied and verified using 
a 4-DoF delta robot. Following a short introduction of the experimental setup 
in Section 4.1, the experimental results and the comparison of approaches are 
presented in Section 4.2. 


4.1 Experimental Setup 


Exemplarily for a typical industrial pick and place robot for highly dynamic ap- 
plications, the delta robot Codian D4-1100 controlled by a standard industrial 
PLC and servo inverters is selected for an evaluation of the proposed methods 
for oscillation reduction. Fig. 2 (a) shows the experimental setup including the 
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acceleration 
sensor I (ir) 


Fig. 2. (a) Delta robot with attached test construction, (b) measurement setup detail. 


whole robot structure. A detailed view of the TCP-mounted test construction 
is shown in Fig. 2(b). To evaluate both methods without further unknown and 
unmodeled effects, a test construction following the mathematical model was 
created. It consists of a flat spring with an exchangeable additional mass, de- 
signed to be mountable to the TCP of the robot, as shown in Figure 1 (b). To 
identify the mechanical parameters of the construction, free oscillations test were 
performed, where the spring stiffness was identified as c = 702 x and the damp- 
ing as d = 0.135 Ms with a measured mass of m = 0.281 kg. Two acceleration 
sensors were used: one rigidly connected to the TCP (sensor I) and the other 
one attached to the mass (sensor II), see Fig. 2 (b). 

The influence of different speeds and different masses was investigated. The agent 
learned its correction a(s) for one motion time T and was also tested for slower 
reference trajectories. Though the agent was trained with the identified mechan- 
ical parameters, it was also tested with varying masses by changing the mass 
attached to the flat spring. In Summary, 10 movements per time and weight case 
were recorded. All following results are also transferable to other structures as 
the approaches are not restricted to a specific hardware or robot type. 


4.2 Experimental Results and Comparison of Approaches 


The variations of motion time T and mass m were performed using four different 
methods for calculating the TCP trajectory u(t): 

— using flatness-based control with all parameters of (9) (“FBC damping”), 

— without damping by using (10) (“FBC”), 

— using the Double DQN approach from Sec 3.2 (“Double DQN”), 

— for comparison purposes, neglecting stiffness and damping (“solid”). 
Fig. 3 shows exemplary acceleration signals for all three methods each compared 
to the solid model for an attached mass of m = 0.281 kg. The highlighted area I 
indicates the time of motion for the TCP and the area II the time after arriving at 
the goal position with remaining mass oscillations. The evaluation concentrates 
on the remaining accelerations after arriving the goal position. Fig. 4 shows 
the results for the time variation tests as box plots. The agent was trained for a 
motion time of 0.4s, mass of 281 g and a linear movement of 0.55 m. The ordinate 
shows the maximal remaining acceleration after the TCP arrives its end position 
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and the abscissa corresponds to the motion time. Note that the axes scales of 
Fig. 4 and Fig. 5 are different, since using the solid model (directly using the 
reference trajectory) leads to higher oscillations. The highest improvement in 
relation to the solid model of around 85% can be seen for the shortest motion time 
in Fig. 4. After investigating different motion times, the influence of incorrectly 
identified masses is examined, for a motion time of 0.4s in Fig. 5. As opposed to 
Fig. 4, different masses attached to the bending beam are given on the ordinate. 
All methods were able to achieve a vibration reduction except for a mass of 98 g, 
for which the solid model performed better. One possible explanation for such 
a behavior is that a small mass tends to vibrate rather than a large one. At the 
same time, the flatness-based control reacts sensitive to deviations of identified 
parameters with respect to the real parameters [14]. 


The experiments indicate that both methods lead to a vibration reduction. 
For the present case, the effect of the spring stiffness is much greater than the 
damping, the flatness-based control approach with consideration of damping was 
just slightly superior to the one without it. 
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Fig. 3. Exemplary acceleration signals in comparison to solid model for (a) Flatness- 
based control with damping, (b) FBC without damping and (c) Double DQN. 
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(a) Mass [g] (b) Mass [g] (c) Mass [g] (d) Mass [g] 
Fig. 5. Variation of the mass with a constant motion time; (a) FBC with damping, (b) 
FBC, (c) Double DQN and (d) solid model. 10 test motions per box. 


The tests with respect to incorrectly identified masses showed similar results, 
however, with deterioration for the lightest mass. Overall, both methods led to 
similar performances in vibration reduction. 


5 Conclusion and Future Work 


This paper illustrates two methods for reducing the vibrations of not rigidly 
coupled objects during highly dynamic robot movements. Especially the object 
vibrations at the end of the movement are observed and can be reduced with 
both methods by up to 85%. This results in a significant increase in placement 
accuracy and a time saving during object manipulation. It can thus be shown 
that both flatness-based control and reinforcement learning with the Double 
DQN are methodically suitable for the aim of vibration reduction of elastically 
coupled handling objects. This clear improvement was also shown with inaccu- 
rately identified object parameters. Both methods are therefore (to a certain 
extent) robust against parameter uncertainties. It is important to mention that 
both methods are completely independent of the considered robot structure, 
since they are based purely on the TCP position and therefore no consideration 
of the robot dynamics has to be carried out. 

Due to the simple underlying linear model of the base-excited mass, there 
are clear advantages for flatness-based control in terms of implementation effort. 
For this reason we consider the flatness-based control approach to be superior 
to the Double DQN, since programming the reinforcement learning algorithm, 
including the time for tuning hyper parameters and training, took much more 
time than designing and implementing the control u(t). However, it should be 
noted that these described advantages decrease with increasing complexity of 
the model. In addition, it is not guaranteed that a flat output can be defined for 
more complex models. In this case, flatness-based control is no longer applicable 
so that the Double DQN approach would be advantageous. In addition, reinforce- 
ment learning offers the possibility to be used in combination with convolutional 
neural networks, enabling it to learn from raw image data [7]. 
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Future work will focus on the comparison of the methods for more complex 
(non-linear) systems regarding the achievable vibration reduction and the im- 
plementation effort. Furthermore, the investigation of adapting the approaches 
to three-dimensional space curves is still an open question. In this paper a value- 
based algorithm of reinforcement learning was used, which is limited to a discrete 
action space. An actor-critic algorithm with a continuous action space promises 
further improvements. 
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Abstract. In previous research, an aerodynamic feeding system was developed, 
which autonomously adapts to different components by using a genetic algorithm 
that controls the physical parameters of the system (e.g. angle of inclination, noz- 
zle pressure). The algorithm starts with two individuals with random values, gen- 
erated within the boundaries of the parameters set by the user. Due to this, the 
setting time - the time that passes until a satisfactory orientation rate is reached - 
is hard to predict. The aim of this work is to identify basic interactions of geo- 
metric component properties with the physical parameters of the aerodynamic 
feeding system to determine in which areas of the workspace a satisfactory solu- 
tion can be expected. By doing so, the initial population of the genetic algorithm 
can be generated based on certain geometric properties and would therefore no 
longer be random, presumably reducing setting time. 

To identify interactions of component properties and system parameters, exem- 
plary components were developed. They represent relevant single properties that 
have significant impact on the aerodynamic orientation process. These compo- 
nents were then fed into the aerodynamic orientation process and their behavior 
was documented. To identify correlations between certain geometric properties 
and physical parameters of the feeding system, the tests were planned and carried 
out using Design of Experiments methods. The results of the tests were also used 
to determine the direct interrelations of said properties and the suitability for aer- 
odynamic orientation. 


Keywords: Design of Experiments, Genetic Algorithm, Aerodynamic Feeding. 


1 Introduction 


In the highly automated assembly processes seen in modern production, part feeding 
plays a vital role. The most common part feeding system used in the industry is the 
vibratory bowl feeder (VBF) [1, 2]. The essential component of a VBF is the bowl with 
a spiral track, which is connected to a drive unit, which creates an alternating rotational 
movement of the bowl using suspension springs and electromagnets [3]. This vibration 
is used to separate parts from the bulk at the bottom of the bowl and transport them up 
the spiral track, where traps are used to either reorient or reject components that are not 
in the desired orientation. While the bowl and the drive components have relatively low 
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costs, the design of the traps and the bowl can be very expensive and time- 
consuming [4]. In order to address this problem, a lot of research was done with the 
aim to decrease the cost for track and trap design using simulation tools [3-8]. However, 
even with simulation tools at hand, retooling of the orienting devices requires manual 
labor in the manufacturing and assembly of the specially designed bowls and traps. 

To meet the demand for a highly flexible and efficient feeding technology, an aero- 
dynamic feeding system has been developed at the Leibniz University Hannover [9- 
12]. The system uses a constant air jet to exert force on the components passing by. 
Using a genetic algorithm, the system is designed to parametrize itself for an optimal 
output rate, so that no manual labor or manufacturing is needed for retooling. The work- 
ing principle of the aerodynamic feeding system and the algorithm for self-parametri- 
zation will be outlined in the following. 


2 The aerodynamic feeding system 


The aerodynamic feeding system presented and used in this work operates with only 
one air jet, which every component passes. In other work, systems have been presented 
that use multiple nozzles or air cushions to orient and transport parts [13, 14]. Fig. 1 
shows the process of aerodynamic orientation in the described feeding system. It can 
be seen that the components behave differently, depending on the orientation they have 
when arriving at the nozzle. Should the workpiece arrive in the wrong orientation, it is 
turned over by the air jet, as can be seen in Fig. 1 a), whereas it keeps its orientation, if 
it already arrives in the correct orientation (Fig. 1 b)). The reason for the different be- 
haviors of the component lies in the shape of the workpiece and the distribution of mass 
in the very same [13]. 

The shape of the workpiece influences the projected inflow area and the drag coef- 
ficient, which determine the lift force transmitted to the workpiece. If the shape of the 
component is not uniform over the entire length, the force transmission also changes 
depending on the position of the component above the nozzle. This results in a rota- 
tional impulse around the center of gravity of the workpiece. The location of the center 
of gravity is determined by the distribution of mass in the component. This also influ- 
ences the dynamic behavior of the part when it is lifted and accelerated by the air flow. 
In addition to their properties, the behavior of the components in the aerodynamic ori- 
entation process is determined by the adjustment of the four physical system parameters 
a, P, v and p seen in Fig. 1. In the exemplary process in Fig. 1, the angle of inclination 
a promotes clockwise rotation of the workpiece and impedes counterclockwise rotation, 
which supports the successful orientation process. The angle £ influences the force of 
gravity acting on the component and determines the impact of the friction between the 
component and the guiding plane. The nozzle pressure p directly affects the magnitude 
of drag force acting on the workpiece. If it is set too low, the component might not 
rotate at all, whereas a higher pressure can lead to multiple and unpredictable rotations. 
Lastly, the component speed v determines, how fast a workpiece passes the air jet and 
therefore how long the drag forces affect it. The parameter can be controlled by adjust- 
ing the speed of a conveyor located ahead of the nozzle. 
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Fig. 1. Illustration of the aerodynamic orientation process [9] 


In order to identify the effects and interrelations of the system parameters on the 
orientation process, BUSCH [15] used Design of Experiments (DoE) methods. In his 
work, BUSCH did not take any geometric or physical properties of the workpieces into 
account, because only the pneumatic plug seen in Fig. 1 was used as exemplary com- 
ponent. Looking at the working principle of the aerodynamic orientation process, it can 
be seen that certain component properties like length, diameter, shape or mass and the 
physical parameters of the aerodynamic feeding system (nozzle pressure, inclination, 
velocity) influence each other. The relevance of understanding the interaction between 
the component properties and the system parameters becomes apparent, when looking 
at the functionality of the genetic algorithm that is used to find a satisfactory set of 
parameters for the feeding process. The goal of the genetic algorithm (GA) is to find a 
set of the four physical system parameters a, f, v and p that induces a satisfactory ori- 
entation rate (e.g. > 95 %). A GA was chosen for this task because the search for a good 
orientation rate constitutes a non-linear optimization problem with a solution space that 
is not necessarily continuous [9, 10, 15]. A major disadvantage regarding the conver- 
gence and predictability of the algorithm is the random initial population since there is 
no input correlating with the component at hand. While this assures that the whole 
workspace of the feeding system is searched for a solution, it also creates a huge 
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solution space. The aim of this work is to determine in which areas of the workspace a 
satisfactory solution can be expected, depending on certain properties of the workpieces 
to feed by identifying basic interactions between geometric component properties and 
the physical system parameters a, p, v and p. 


3 Exemplary components 


To identify the interactions of component properties with the physical system para- 
meters, these properties have to be defined first. The properties are divided into physical 
and geometrical features. Geometrical features include length, width, height, shape, 
symmetry and center of gravity. Relevant physical properties are mass, moment of in- 
ertia, damping properties and friction with other materials. Since it is not possible to 
determine the interactions of all of those properties in this work, three geometric prop- 
erties are selected, that can be varied independently and determined easily. The selected 
properties are: 


e Length 
e Diameter 
e Eccentricity of center of gravity 


The independence of the component properties is an important factor for the DoE 
method because it allows a design where each experiment represents the change of only 
one variable without altering the others. As aforementioned, the DoE method was al- 
ready used in prior work by BUSCH [15] to identify the effects and dependencies of 
the system parameters on the orientation of a pneumatic plug. The geometric properties 
were not taken into account, because only one type of plug was used. In this work, DoE 
is used to evaluate the effects and dependencies of geometric components properties in 
order to make the results more applicable to different types of workpieces. 

The exemplary part used for the trials has to represent the selected properties indi- 
vidually, with as little interferences as possible. The simplest part that can be described 
with length and diameter is a cylinder. The parameters length and diameter can be var- 
ied independently. To achieve eccentricity of the center of gravity of a cylindrical part 
a hole from one of the front surfaces is included in the design. The eccentricity can be 
varied by altering the depth and diameter of the hole. At the same time, the hole does 
not affect the outer shape of the cylinder. Fig. 2 shows a drawing of the component, 
labeled sleeve in the following, with the parameters / (length) and d (diameter). 


Fig. 2. Sketch of the sleeve used as exemplary component 
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In order to set up a test plan, it is necessary to define the boundaries of the parameters 
l and d. Suitable values for the eccentricity £ also have to be selected. Preliminary tests 
show that it is very difficult to make a universal statement about the minimum and 
maximum values a component’s property must have to be suitable for aerodynamic 
orientation. The reasons for this are: 


e Interactions between separate component properties 
e Uncontrollable behavior of the components before and after the orientation process 
e Constructional characteristics of the feeding system 


An example for the interaction between separate properties are the length and diameter. 
The boundaries of each of those properties depend, to some extent, on each other. For 
example, preliminary testing showed that a sleeve with a length of 12 mm can be ori- 
ented, ifthe diameter is not bigger than 6 mm. Generally the ratio of length and diameter 
should be greater than two to allow aerodynamic orientation and to avoid uncontrolled 
behavior of the component (e.g. component tumbling down the guiding edge). Another 
limiting factor are the constructional characteristics of the feeding system. For example, 
the maximum flow and therefore the maximum impulse of the air jet is limited by the 
diameter of the air nozzle. The pressure of the air jet can only be increased until the 
critical pressure ratio J is reached. After that, the jet is no longer usable for technical 
applications [13, 14]. For air, the critical ratio is 0.528, which limits the nozzle pressure 
p to 0.89 bar. 

Different versions of the exemplary sleeve were printed using a 3D-printer and then 
fed into the orientation process. A part was evaluated as generally suitable for aerody- 
namic feeding, if the constant air jet could reorient it. The measurements of the resulting 
exemplary components for the following experiments are presented in Table 1. The 3D- 
printed components are shown in Fig. 3. 


Table 1. Measurements of exemplary components 


Component No. length Zin mm diameter din mm eccentricity Zin % 
1 24 6 5 
2 150 6 5 
3 24 12 5 
4 150 12 5 
5 24 6 10 
6 150 6 10 
7 24 12 10 
8 150 12 10 


336 


SH, 


Fig. 3. 3D-printed exemplary components (sleeves) with component numbers (cf. Table 1) 
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4 Interactions of component properties and system parameters 


To investigate the interaction of component properties and system parameters, an ex- 
perimental plan has to be designed. In addition, the target figures of the trials have to 
be defined. After that, the trials are carried out and the results are evaluated. 


Experimental plan. To design a proper experimental plan, it is necessary to first de- 
termine the input and output parameters of the trials. In the previous section, length, 
diameter and eccentricity ofthe components used were selected as component param- 
eters for this work. Additionally, the physical parameters a, 2, p and v are defined as 
input parameters. 

The aim of the experiments is to find out how to narrow the limits of the physical 
system parameters depending on the selected geometric component properties length, 
diameter and eccentricity in order to achieve a high orientation rate. The orientation 
rate is therefore defined as the only output. It is easy to quantify and has distinct limits 
(0 % to 100 %). Furthermore, a high orientation rate is the optimization goal of the 
genetic algorithm, the improvement of which is the goal of this work. 

In the previous section, a full factorial test plan for the geometric properties of the 
components was already drawn up and corresponding values for the upper and lower 
limits selected through preliminary testing. In order to determine the interactions of the 
parameters a, J, p and v and the component properties with the orientation rate, it is 
necessary to include the system parameters in the test plan. Table 2 shows the values 
of the parameters (factors) a, £, p and v for each of the components. 

The upper and lower factor settings (Xmin / Xmax) of the physical system parameters a, 
f and v are determined by the constructional boundaries of the feeding system. As men- 
tioned before, the boundaries of the pressure p are 0 bar and 0.89 bar. Since the com- 
ponents have different weights, as a result of the varying dimensions, the limits for this 
parameter cannot be generally determined for all components. If the pressure is too low, 
a component will not be lifted at all, if the pressure is too high, the component will spin 
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multiple times, uncontrollably or even be blown out of the feeding system. To avoid 
this, additional preliminary tests were carried out to determine the boundaries of the 
pressure p. 


Table 2. Factor setting for the physical system parameters 


Comp. Omin in Omax in Bmin in Bmax in Vmin in Vmax in Pmin in Pmax in 
o o o 


No. 2 m/min m/min bar bar 
1 20 24 40 50 55 75 0.05 0.1 
2 20 24 40 50 55 75 0.3 0.4 
3 20 24 40 50 55 75 0.5 0.6 
4 20 24 40 50 55 75 0.8 0.89 
5 20 24 40 50 55 75 0.05 0.1 
6 20 24 40 50 55 75 0.3 0.4 
7 20 24 40 50 55 75 0.5 0.6 
8 20 24 40 50 55 75 0.8 0.89 


The use of a full factorial test plan with four two stage factors would result in a test 
effort of 16 test runs per workpiece. Accordingly, a total of 128 test runs would have to 
be carried out. Due to the size and variety of the workpieces, the attached centrifugal 
feeder and the automated recirculation system cannot be used. Therefore, the compo- 
nents have to be applied to the conveyor (parameter v) manually. In order to reduce the 
testing effort, the full factorial test plan is reduced to a screening plan presented in Table 
3. In order to reduce the influence of environmental conditions (temperature, humidity), 
the screening plan was run twice in the interval of two weeks. The order of the individ- 
ual test runs (8 runs per workpiece) was randomized. For each test run, 100 workpieces 
are fed into the orientation process in alternating initial orientation. This means that if 
no reorientation takes place, the orientation rate is 50 %. If all workpieces that arrive at 
the nozzle in wrong orientation are reoriented, but none of the workpieces arriving at 
right orientation are reoriented, the orientation rate is 100 %. 


Table 3. Screening test plan for each component 


Test run No. a £ v p 
1 z E = E 
2 + . - + 
3 - + - + 
4 + + - - 
5 - x + + 
6 + x + 
7 - + + š 
8 + + + + 


Results. The results of the two screening trials show a high consistency regarding the 
orientation rates measured for each parameter configuration. Since the presentation of 
the results of all experiments would go beyond the scope of this work, the analysis of 


338 


the test results is presented in the following. In order to determine the influence of a 
system parameter on the orientation rate O, the main effects of each parameter have to 
be calculated. Table 4 shows the main effects of all components together with the spe- 
cific properties of the components. For example, the main effect E, for the second com- 
ponent indicates that increasing the velocity v from 55 to 75 m/min is associated with 
an average increase of the orientation rate of 15.9 %. To calculate the effect of the 
parameter v for the first component, the results of the 16 test runs (two screenings with 
eight test runs each) done with the component are split in two halves, one representing 
the test runs with high velocity v and the other representing those with a low velocity. 
The effect is then calculated by subtracting the mean orientation rate of the half with 
lower velocity from the mean orientation rate of the half with higher velocity (equation 
1). In addition to the main effects, secondary effects - interactions between the factors 
- play a role, but will not be discussed in more detail here. 


E; comp a ne re (1) 


Table 4. Main effects of the physical parameters on the orientation rate 


Comp. No. /inmm dinmm Zin% Eain% Ein% E,in% Epin% 


1 24 6 5 6.6 3.6 4.9 0.4 
2 150 6 5 11.4 -5.6 15.9 -3.6 
3 24 12 5 -7.5 10.3 -4.3 -1.5 
4 150 12 5 -11.4 -10.4 1.9 8.4 
5 24 6 10 1.1 1.4 4.6 -3.1 
6 150 6 10 14.0 -14.3 7.3 -8.0 
7 24 12 10 -8.1 9.6 -4.1 -2.9 
8 150 12 10 -2.3 -15.8 -23.8 10.3 


Derived from the analysis of the main effects and the secondary effects, recommen- 
dations for the selection ofthe system parameters can be made. These are given in Table 
5. In the table, a “+” represents a high value of the component property or system pa- 
rameter and a “-”represents a low value. For example, the combination “/ -”, “d +” and 
“E +” represents a component with a small length, a high diameter and a high eccen- 
tricity of the center of gravity. For such a component, a low inclination angle a, a high 
inclination angle 2 and a high velocity v should be selected to achieve higher orientation 
rates. It is expected that by narrowing the range of those parameters in an area, where 
high orientation rates are expected to be more likely, the setting time of the system can 
be reduced. The pressure p has to be considered with caution, as the upper and lower 
limits of the pressure are not constant between the different components in contrast to 
the limits of the other three system parameters. 
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Table 5. Recommendation for the selection of system parameters dependent on component 


properties 
l d ra a ß v p 
a i ; + 
T ae = + = 
- - i + - 
H - - - + 


5 Interaction of component properties and orientation rate 


In the previous section, interactions between component properties and system param- 
eters were investigated. In the following section, the results of the tests carried out are 
further analyzed to find out how the properties of the components and the orientation 
rate are related. The results will be used to help predict the suitability of a workpiece 
for aerodynamic orientation. 

In order to evaluate the influence of the component properties length, diameter and 
eccentricity, again, a quality characteristic must be selected. As described in section 4, 
the orientation rate is the most suitable characteristic to determine the suitability for 
aerodynamic orientation. In the tests, orientation rates were determined for 64 different 
combinations of component properties and parameter sets. From the results of these 
tests, different variants of the quality characteristic orientation rate O can be derived. 
On the one hand, the mean value of the orientation rate for each exemplary component 
Onean is selected as quality characteristic. The value indicates the average orientation 
rate that is achieved with one component. On the other hand, the maximum value, which 
indicates the highest orientation rate accomplished with an exemplary component Omax 
is selected. The values are given in Table 6. 


Table 6. Mean and maximum orientation rate of 16 test runs for each component 


Comp. No. Lin mm din mm Zin% OmeanriNn% Omax in Yo 
1 24 6 5 57,6 66,0 
2 150 6 5 65,3 92,0 
3 24 12 5 43,0 57,5 
4 150 12 5 61,1 94,5 
5 24 6 10 73,3 79,0 
6 150 6 10 56,6 90,0 
7 24 12 10 43,8 57,0 
8 150 12 10 66,4 98,0 


The values in Table 6 are used to determine the effects of the individual component 
properties on the orientation rate. To do so, the main effects of parameters and their 
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interrelations are calculated analogously to section 4 (cf. equation 1). These factors are 
calculated using the mean orientation rate of all test runs with one component as quality 
criteria on the one hand and the maximum orientation reached with a component on the 
other hand. The results are illustrated in Fig. 4, sorted by size of the effects. It can be 
seen, that the factors have similar values, regardless of whether they are calculated from 
the mean or maximum values of the orientation rate in Table 6. 


35,0 
30,0 
25,0 
20,0 

- 9 / 
15,0 12,403 


10.0 —7,9 i 7,2 
4,8 3335 
5,0 > 3,3 
6.0 N l Fa . == a 
-5,0 E a ),2 


-10,0 
-15,0 


28,8 


Effect in % 


Eı Era Ea Eraz Ez Ez Eaz 
Factor 


E based on mean value E based on maximum value 


Fig. 4. Visualization of the effects of component properties and their interrelations (factors) on 
the orientation rate sorted by size 


The results indicate that the length of a component has a large positive impact on the 
orientation rate. With an effect of 7.9 % calculated from the average orientation rate 
and 28.8 % calculated from the maximum orientation rate, the length has the largest 
effect of the three properties investigated in this work. Therefore, it can be concluded 
that longer components are potentially more suitable for aerodynamic orientation than 
shorter components. This may be because, on the one hand, a longer component is in 
the air jet for a longer period of time, when moving at the same velocity as a smaller 
component and is therefore more strongly influenced by the air jet. On the other hand, 
the lever with which the lifting force of the air jet rotates the component around the 
center of gravity becomes longer. It has been observed in the preliminary tests that 
longer components tend to have a more stable movement behavior on the guiding plane 
and edge. 

The second largest effect is the two-factor interaction of length and diameter, which 
is positive. That indicates that components with a very large or a very small length to 
diameter ratio are less suitable for the orientation process. The ratio of the components 
used in the test runs varies from //d = 2 to l/d = 25. From the results illustrated in Fig. 
4 it can be deduced that both extreme characteristics achieve a lower orientation rate in 
comparison to the components with ratios like //d = 4 or l/d = 12.5. 
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The effect of the diameter is negative, which indicates that the orientation rate di- 
minishes as the diameter of the component increases. This could already be observed 
in the preliminary testing and is attributed to the Bernoulli Effect, which reduces the 
lifting force applied to a component. When the sleeve passes over the nozzle, it creates 
a gap between itself and the nozzle guiding edge. Therefore, the flow path of the air jet 
is narrowed which results in a higher velocity of the fluid. This, according to Bernoulli’s 
principle, leads to a decrease of pressure between the sleeve and the guiding edge ulti- 
mately resulting in the part being dragged down [14-16]. This effect becomes stronger 
with increasing component diameter. 

The eccentricity of the center of gravity has a positive influence on the orientation 
rate. This has already been investigated by LORENZ [13] and is confirmed by the tests 
carried out for this work. An interesting observation concerns the interaction of eccen- 
tricity and length, which is negative. From this it can be concluded that an increase of 
the eccentricity has a higher positive impact on the orientation rate for short compo- 
nents than for long components. The two-factor interaction of eccentricity and diameter 
is very small and therefore not further discussed. 

The three-factor interaction of length, diameter and eccentricity is positive. The In- 
terpretation of this interaction is difficult because for some combinations of geometric 
properties it contradicts the other effects. For example, according to the three-factor- 
interaction, a long component with a high eccentricity of the center of gravity must 
have a high diameter to reach a high orientation rate. This contradicts the clearly nega- 
tive effect of the diameter. At the same time, the three-factor interaction can be used to 
determine what a suitable component should look like. It is difficult to make a general- 
ized statement about the geometric properties of a suitable workpiece, since the suita- 
bility is also influenced by factors not considered in this work, like nozzle diameter and 
shape. Yet, the results obtained by the analysis of the test data indicate that workpieces 
with a higher length (cf. E) and eccentricity (cf. Ez) as well as a medium sized diameter 
(cf. Era and Eraz) are more suitable for aerodynamic orientation in the sense that a high 
orientation rate can be reached at different parameter configurations. 


6 Conclusion and Outlook 


The aim of this work was to determine the interaction between component properties 
and the setting of the physical parameters of the aerodynamic feeding system. In addi- 
tion, it was to be found out how the change of individual component properties affects 
the suitability for aerodynamic orientation. 

In order to achieve these objectives, relevant component properties were defined in 
a first step. By conducting preliminary tests, suitable sample components were designed 
and subsequently manufactured using a 3D printer. Based on eight sample component 
geometries, a screening test plan was drawn up in order to be able to represent the 
influencing factors well. A total of 128 test runs were carried out with 16 test runs per 
component geometry. In each test run, 100 components were fed to the aerodynamic 
orientation process and the resulting orientation rate was measured. 
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By evaluating the test results it was possible to create a matrix which makes it pos- 
sible to generate recommendations for the setting of the physical system parameters a, 
£. v and p, depending on given component properties such as length, diameter and ec- 
centricity of the center of gravity (cf. Table 5). The findings can be used to narrow the 
range of the system parameters in order to reduce the search space and therefore the 
setting time of the genetic algorithm. Furthermore, positive and negative effects of cer- 
tain component properties on the orientation rate and thus on the general suitability for 
aerodynamic orientation could also be determined (cf. Fig. 4). A higher suitability 
means that a high orientation rate can be achieved easier, which in return also decreases 
the setting time of the genetic algorithm. 

The interactions and correlations derived from the tests carried out in this work are 
based on the aforementioned set of 128 test runs (two screenings with 64 test runs each). 
The goal of future work is to increase the accuracy of the recommendation matrix by 
repeating the test to reduce the influence of statistical errors. Also, the tests should be 
carried out using a different test plan. Using the screening test plan, it was assumed, 
that the interrelations are linear in between the boundary values. One reason for using 
a genetic algorithm for setting the system parameters is the non-linearity of the optimi- 
zation problem. Therefore, to better model the interrelations in the entire workspace of 
the aerodynamic feeding system, a new test plan with a Central-Composite-Design 
(CCD) will be generated and carried out. The results of the CCD test plan will also be 
used to refine and validate the interrelation of component properties and suitability for 
aerodynamic orientation as shown in section 5. 

Another objective of future work will be to investigate if the results generated with 
the exemplary components in this work are applicable to components of entirely differ- 
ent geometry. The sleeves used in this work were designed to only represent the effect 
of an eccentric center of gravity, length and diameter on the orientation process. With 
the cylindrical outer contour, a very simple geometry was chosen. The aim of future 
work must therefore also be to reproduce the interrelationships of the component prop- 
erties with the orientation process for more complex components. Two further compo- 
nent geometries have already been developed for this purpose and can be seen in Fig. 5. 
With those components, the success of the aerodynamic orientation process is not only 
based on the eccentricity of the center of gravity but also on the varying projected in- 
flow surface of the components. 


VAAR NARI 


Fig. 5. 3D-printed exemplary components for further testing 
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